Asked in: healthtech LLM interviews; safety / responsible-AI roles.
The L4 answer is generic LLM deployment with extra logging. The L6 answer treats the clinician as a first-class part of the system and designs around regulatory, audit, and harm-floor requirements.
What an L4 answer sounds like
“Add disclaimers, log everything, and have a human review the outputs.”
This is a starting point, not an answer. It misses the regulatory framing, the workflow integration, and the operational discipline. You’ve heard the buzzwords.
What an L5 answer sounds like
“Healthcare has three additional constraints:
- Regulatory: HIPAA for data, FDA for medical-device-classified software, GDPR/local equivalents.
- Harm floor: a hallucinated diagnosis or treatment recommendation can directly harm patients.
- Workflow: clinicians are the decision-makers. The system supports them; it doesn’t replace them.
Architecture choices that follow:
Scope tightly. Use LLMs for clearly bounded tasks: documentation drafting, summarization of patient records, retrieval of guidelines, structured data extraction. Avoid open-ended diagnosis or treatment recommendation in v1.
Ground in trusted sources. RAG over verified medical literature, clinical guidelines, and the patient’s own records. Refuse to answer from model knowledge alone for clinical questions.
Citations are mandatory. Every clinical claim must trace to a source the clinician can verify in seconds.
Confidence-aware UI. Surface uncertainty explicitly. Distinguish ‘high-confidence summary’ from ‘best-effort interpretation’. Make refusal a visible, normal output.
Human-in-the-loop by default. No clinical action triggered by LLM output without clinician confirmation. The LLM drafts; the clinician decides.
Audit trail end-to-end. Every input, retrieval, generation, citation, clinician acceptance/edit logged for review.”
This is L5. You’ve named the constraints and let them drive the architecture.
Learning objective
Trace where evidence, clinician authority, and versioned audit data prevent an LLM draft from becoming an autonomous clinical action.
flowchart TB
accTitle: A healthcare LLM supplies traceable evidence to a clinician, not an autonomous clinical action
accDescr: Intended use and regulatory classification set a bounded task. Minimum-necessary patient context and verified clinical sources feed an LLM that returns a structured draft with claim-level citations and uncertainty or refusal. A clinician independently reviews the source basis and patient fit. Unsupported or out-of-scope output is rejected or edited and triggers no clinical action; accepted output becomes the clinician's decision in the existing clinical workflow. Inputs, retrievals, outputs, model version, clinician edits or acceptance, and downstream outcomes enter an audit record used for workflow evaluation, fairness slices, incident response, rollback, and controlled updates.
Scope{"Intended use + regulatory class<br/>What may this system support?"}
Evidence["Minimum-necessary patient context<br/>+ verified clinical sources"]
Draft["Bounded LLM draft<br/>claims + citations + uncertainty / refusal"]
Review{"Clinician independently reviews<br/>source basis + patient fit"}
Stop["Reject, edit, or refuse<br/>no clinical action"]
Decide["Clinician confirms<br/>the clinical decision"]
Workflow["Existing clinical workflow<br/>document · communicate · act"]
Trace[("Versioned audit record<br/>input · retrieval · output · clinician decision · outcome")]
Operate["Workflow eval + fairness slices<br/>incident response · rollback · controlled update"]
Scope --> Evidence
Evidence --> Draft
Draft ==>|"draft only"| Review
Review -->|"unsupported"| Stop
Review -->|"supported"| Decide
Decide --> Workflow
Evidence -.->|"provenance"| Trace
Draft -.->|"model + citations"| Trace
Stop -.->|"edit / reject"| Trace
Decide -.->|"confirmation"| Trace
Workflow -.->|"outcome"| Trace
Trace --> Operate
Operate -.->|"gate update / rollback"| Scope
class Scope,Review viz-focus
class Evidence viz-input
class Trace viz-state
class Decide,Workflow viz-output
class Stop viz-warning
class Scope viz-tall,viz-wide
Read it this way: follow the solid path first. Regulation and intended use bound the task; verified evidence supports a structured draft; and only a clinician who can inspect the source basis and patient fit can turn that draft into a clinical decision. Then follow the dashed paths: every refusal, edit, confirmation, and outcome joins the same versioned trace for evaluation, incident response, rollback, and controlled updates. The gate labels and arrow styles carry the meaning without color. Original schematic informed by the FDA Clinical Decision Support Software guidance, the WHO guidance on AI for health, and the NIST AI Risk Management Framework.
What an L6 answer adds
“…practical operational considerations:
Eval is harder than usual. Generic LLM benchmarks are useless; medical benchmarks (USMLE-style, MedQA) test knowledge but not safety. Build evals from real clinical workflows, scored by clinicians, with explicit failure-mode tracking. Pre-launch eval should include adversarial cases (prompts likely to elicit dangerous outputs) and edge cases the team’s clinical advisors flag.
Refusal-rate is a metric, not a failure mode. A model that refuses appropriately on out-of-scope clinical questions is better than one that confidently makes things up. Track refusal rate per category; tune to balance helpfulness against the harm floor.
Bias and fairness are first-order. LLMs encode biases from training data; in clinical contexts these can produce demographic disparities in care. Slice eval by demographic dimensions; gate launches on no-regression.
Regulatory categorization drives the build. If the system is FDA-classified Software as a Medical Device (SaMD), the validation, documentation, and change-management requirements are enormous. Many teams scope explicitly to non-SaMD use cases (admin, documentation) to avoid this.
Incident response is part of the architecture. When (not if) something goes wrong, you need: traceable logs, rapid revert, communication path to affected clinicians and patients, regulatory disclosure procedures. Build the response plan before launch.
Limit model autonomy through the prompt and output structure. Output is structured (e.g., ‘suggested differential’ rather than free prose), with defined fields, defined refusal language. Constrains both the model’s failure modes and the clinician’s expectations.”
Tells that get you a strong-hire vote
- You scope tightly (admin tasks, documentation, structured extraction; avoid open-ended diagnosis in v1).
- You name HIPAA, FDA SaMD explicitly and let them drive scope.
- You make clinician-in-the-loop the default architecture.
- You treat refusal as a feature, not a failure.
- You bring up fairness slices and incident response.
Tells that get you down-leveled
- Suggesting an LLM diagnoses or treats patients in v1.
- Adding “disclaimers” as the safety strategy.
- No regulatory awareness.
- No mention of bias or demographic slicing.
Common follow-up
“How would you handle a clinician who consistently overrides the LLM’s correct outputs?”
The L6 answer:
“Two angles. From the system perspective: log the override pattern, surface to the clinician their override-vs-system-correctness rate, and analyze the disagreement pattern (is the clinician right and the system wrong, or vice versa, or are they disagreeing on judgment calls?). From the human perspective: clinicians are the responsible parties; their judgment overrides the LLM by design. The system’s job is to make their decision easier and traceable, not to coerce. If the override rate is consistently high, that’s a signal the system isn’t useful to that clinician, not that the clinician needs to change.”
Related: How do you handle hallucinations in production?, LLM Evals essay, RAG for legal documents.