Consistency & Reliability at Scale

What you'll learn

A prompt that works on your three test inputs is not finished. In production it will meet inputs you never imagined — empty, enormous, malformed, in the wrong language, or actively weird. Reliability is the craft of making a prompt behave sensibly across all of them, and failing safely when it can't.

By the end of this module you'll be able to:

• See the gap between "works once" and "works every time"
• Design prompts that hold up across the full variety of real inputs
• Handle empty and malformed input before it reaches the model
• Detect silent failures and recover with retries and fallbacks

Works once vs. works always

A demo runs the happy path once. A product runs the same prompt thousands of times against inputs from real people — typos, pasted HTML, other languages, empty fields, messages designed to break it. The question shifts from "did it work?" to "what fraction of the time does it work, and what happens the rest of the time?"

Designing for varied inputs

Build robustness into the prompt itself:

• Cover the edges in your instructions. Say what to do with missing fields, multiple items, or irrelevant input.
• Give an explicit "none" path (Module 5): what to output when there's nothing to extract.
• Test on a diverse set, not three friendly examples — include the messy and the empty.
• Constrain the output (Module 14) so downstream code can always parse it, even on a bad input.

Empty & malformed input

The most common production surprise is the emptiest one: a blank field, whitespace, or nonsense. Handle these before spending a model call, and tell the model explicitly what to do when the content isn't what it expects.

Detecting silent failures

The dangerous failures aren't crashes — they're confident, wrong-but-well-formed answers that sail through unnoticed. Guard against them:

• Validate shape and content, not just JSON syntax (Module 14) — right fields, sane values, in range.
• Add sanity checks in code: a date in the past, a total that doesn't add up, an enum value you don't recognize.
• Ask for a confidence signal when useful, and route low-confidence cases to a human or a heavier method.
• Log and sample outputs so you notice quality drift before your users do.

Retries & fallbacks

When validation fails, don't crash — recover. Retry (sometimes a second sample simply succeeds), and if it still fails, fall back to a safe default and flag it. This defensive loop turns an occasional bad response into a handled edge case instead of an outage.

def extract(text):
    if not text or not text.strip():
        return {"ok": False, "reason": "empty input"}

    for attempt in range(3):                 # retry on failure
        raw = call_model(prompt_for(text), temperature=0)
        try:
            data = json.loads(raw)
            if valid(data):                  # validate shape + content
                return {"ok": True, "data": data}
        except json.JSONDecodeError:
            pass                             # fall through and retry

    return {"ok": False, "reason": "no valid output"}   # graceful fallback

Recap & quick check

Your prompts are now structured, controlled, and dependable. But reliability has a ceiling when the model lacks the facts — so next we ground it in real knowledge. Next up: Phase 5, Module 18 — Why Models Hallucinate, and How to Ground Them.