A polished AI demonstration does not tell you how an engineer handles unreliable inputs, ambiguous quality targets or a failing release. An interview for a production LLM role should explore those decisions. Start with a shared scenario rather than a quiz about tool names.
The questions below are suggested prompts, not a validated test. Adapt them to the work and seniority of the role, and tell candidates which assumptions they can make.
1. How would you decide whether this feature is good enough?
Describe a product, such as an assistant that answers questions about internal documentation. Ask the candidate to propose an evaluation approach before discussing implementation. What does a useful answer look like? Which failures matter most? Who can judge whether the output is acceptable?
Listen for representative examples, explicit quality criteria and a distinction between a quick development check and a release decision. Follow up with: “What would make your evaluation misleading?”
2. Users report confidently wrong answers. Where do you start?
Give the candidate room to gather information. Would they inspect the user question, retrieved material, model response and version history? Can they separate a missing source from a retrieval problem or a generation problem?
A useful answer should offer an investigation sequence and explain what evidence would change the diagnosis. Do not require the candidate to guess a hidden root cause. The assessment is about reasoning under uncertainty.
3. The feature is too slow. What do you measure?
Ask how they would establish where time is spent and what experience the user needs. Discuss whether the relevant measure is the time until an initial response, the time until completion, or another product-specific event.
Then ask what trade-offs they would test. Look for a plan to measure quality and cost alongside responsiveness. A candidate should not have to recite a favourite optimisation to demonstrate sound performance judgement.
4. How would you release a change safely?
Introduce a change to the model, prompt or retrieval pipeline. Ask how the candidate would compare behaviour, monitor the release and recover if it performs poorly. Who makes the release decision? What information is needed to investigate a user report?
Keep the discussion proportionate. A small internal tool and a customer-facing system with strict operational requirements may need different controls.
5. What did you personally own?
Invite the candidate to explain a real project using anonymised details. Ask about a decision that did not work, what they learned and how the team changed course. Clarify which parts they implemented and where they collaborated with others.
Combine this conversation with a small, role-relevant task where needed. Use the technical interview scorecard to keep the final decision tied to the role rather than interview fluency.