When would you fine-tune vs prompt vs RAG?

Choose how to repair a model service that uses an outdated policy and violates its required output format.

Problem

Context. A fictional support service must follow an authorized policy revision and return a defined JSON response. Its baseline fails both policy correctness and output-format checks.

Task. Design tests that isolate those failures, then choose among prompting, retrieval, fine-tuning, or a justified combination. Specify success measurements and rollback conditions.

Constraints. Use only authorized policy evidence. Separate changing facts from learning behavior or enforcing an exact output contract. Compare operational cost and freshness needs without assuming one intervention fixes both failures.

A fictional support service must apply yesterday’s authorized policy revision B and return two JSON fields. Policy A allowed refunds within seven days; B allows thirty days. The request concerns day fourteen. The baseline applies A and returns malformed JSON.

Choose tests that separate missing policy evidence from output-format failure. Then decide what to change, what to measure and how to revert.

Picture: Two Failed Checks, Not Three Ranked Techniques

The current-policy check expects eligibility under B. The format check expects a boolean eligible and a policy_version string, with no other fields. Passing either check does not imply passing the other.

Use this stipulated diagnostic ledger; these are example outputs, not measured model runs.

Supplying known-correct context bypasses retrieval to test whether the model can use the needed evidence. It does not prove a retrieval system will find B. If the model still misreads B, better retrieval alone cannot resolve that failure.

Make the Decision Inspectable

These interventions compose; they are not mandatory steps in a ladder. Training can learn facts, and retrieved text can change behavior. The categories identify where the system changes, not exclusive effects. Keep the complete release contract fixed when comparing candidates.

Change 1: Request Volume Changes the Economics

New task. A prompt adds 0.002 currency units per request. Tuning costs 2,000 initially and reduces that increment to 0.001. Assume equal quality and exclude costs common to both systems.

Picture and change. Accumulate each variable cost, then add fixed training and separate refresh costs.

from decimal import Decimal


def adaptation_costs(requests, refresh_cost=Decimal("0")):
    if requests < 0 or refresh_cost < 0:
        raise ValueError("cost inputs must be nonnegative")
    prompt_cost = Decimal("0.002") * requests
    tuned_cost = Decimal("2000") + refresh_cost + Decimal("0.001") * requests
    return prompt_cost, tuned_cost


assert adaptation_costs(2_000_000) == (Decimal("4000"), Decimal("4000"))

Check. The simple break-even is two million requests, not two million tokens. A 500-unit refresh moves it to 2.5 million requests.

Use when. Measured recurring savings can repay adaptation before the model or task changes.

Change 2: The Task Is an Exact Grammar

New task. Accept a whole identifier ACCT- followed by six ASCII digits. Preserve leading zeros and reject surrounding text.

Picture and change. Match the declared grammar instead of sampling a generated answer.

import re


def parse_account_id(text):
    matched = re.fullmatch(r"ACCT-([0-9]{6})", text)
    return matched.group(1) if matched else None


assert parse_account_id("ACCT-000123") == "000123"
assert parse_account_id("ACCT-000123\n") is None

Check. Syntax validity does not establish that an account exists or that the caller may access it.

Use when. A parser, classifier or task-specific model meets the actual requirement more predictably.

Failure: Training Becomes a Freshness Strategy

A weekly-changing policy needs source/version maintenance even if the model once learned it. Likewise, finding B does not ensure correct interpretation or safe handling of untrusted content.

Transfer Check

Could one system use fine-tuning, retrieval and prompting together?

Reasoned Answer

Yes. A tuned model can receive current authorized evidence and request-specific instructions. Evaluate the combined system; neither parameter updates nor citations remove schema, access or semantic checks.