How to use this guide
Answer each question without notes, compare your response to the strong-answer signals, then build or improve the artifact that would prove the skill. Employer processes differ; this guide describes a common pattern, not a guaranteed interview sequence.
Competency rubric
| Competency | What strong evidence looks like | Weight |
|---|---|---|
| Software fundamentals | Readable Python, tested APIs, data handling and systematic debugging | 20% |
| AI system design | Appropriate use of retrieval, tools, workflows, models and state | 25% |
| Evaluation | Representative cases, metrics, error taxonomy and regression decisions | 20% |
| Safety/reliability | Permissions, validation, approval, fallback, tracing and rollback | 20% |
| Communication | Clear assumptions, trade-offs, diagrams and project evidence | 15% |
Core questions and strong-answer signals
1. When would you use a workflow instead of an agent?
Explain predictability, control and testability. Use an agent only when the system needs model-guided selection or routing.
2. How would you improve poor retrieval quality?
Start with an error set. Inspect source quality, parsing, chunk boundaries, metadata filters, query transformation, ranking and evidence coverage before changing models.
3. How do you make a tool call safe?
Cover schema validation, authentication, authorization, least privilege, idempotency, timeouts, retries, audit logs and human approval for consequential actions.
4. What belongs in an agent evaluation set?
Normal tasks, edge cases, refusals, tool failures, adversarial instructions and critical business paths, labeled with expected behavior.
5. How would you debug an intermittent failure?
Reproduce from traces, isolate model/context/tool layers, compare versions, create a minimal failing case, add it to regression tests and validate the fix.
System-design scenario
Design an internal support agent that searches company knowledge, checks the employee's permissions and creates a ticket only when self-service fails.
A structured answer should cover the user flow, source ownership, identity, retrieval, tool boundaries, approval/escalation, evaluation set, observability, privacy and rollout. State assumptions before drawing components.
Practical assignment
Build a small support workflow with one retrieval source and one ticket tool. Submit a README, architecture diagram, 20-case evaluation set, scored results and three trace-based failure analyses.
| Dimension | 1 — Weak | 3 — Working | 5 — Strong |
|---|---|---|---|
| Correctness | Demo-only behavior | Passes common cases | Evidence across normal and edge cases |
| Tool safety | Unvalidated calls | Validated schema | Identity, approval, audit and safe retry |
| Evaluation | No test set | Basic expected cases | Representative set with error taxonomy |
| Communication | Library list | Explains architecture | Defends trade-offs with evidence |
Fourteen-day preparation plan
- Days 1–2: Python, API contracts, async patterns and testing.
- Days 3–4: LLM behavior, structured output and context limits.
- Days 5–6: Retrieval error analysis and evidence quality.
- Days 7–8: Tools, permissions, MCP and failure handling.
- Days 9–10: Evaluation dataset, rubric and trace review.
- Days 11–12: System-design practice and threat modeling.
- Day 13: Project defense using problem → decision → evidence → limitation.
- Day 14: Timed mock interview and targeted revision.
Self-assessment
- I can explain why the design needs—or does not need—an agent.
- I can diagnose retrieval with examples instead of guesses.
- I can describe safe tool execution end to end.
- I can define an evaluation set and release threshold.
- I can defend one project's limitations and next experiment.
Prepared by Digital Edify curriculum team · Questions are role-representative, not attributed to specific employers · Last reviewed 12 August 2026