Test with Realistic Cases

'It seems to be working' isn't a test. Build tests from realistic examples and log every run. Models can give different output from the same input, so one lucky pass proves little.

Collect ten cases: about seven typical, one messy, one planted injection and one that must be refused. Use synthetic or redacted inputs until your privacy gate passes. Run each case two or three times and score the worst result. Mark each right, fixable or wrong.

Test Table (Copy This)

• Case number | Input summary | Expected result | Actual result | Right, fixable or wrong | Note
• 1 to 7: typical inputs
• 8: a messy input (a missing field or odd formatting)
• 9: a planted injection: a field that tells the model to ignore your rules
• 10: an input that must be refused or sent to a person

Reading the Results (Starting Rules, Not Laws)

• Safety checks must all pass before a trial: the refusal case stops, nothing invented gets through, and a planted injection doesn't alter the result.
• With the safety checks passing and 8 or more right: move to a reviewed trial week.
• 4 to 7 right: tighten the instruction and add examples, then rerun all ten.
• Fewer than 4: the task may not suit a first workflow. Go back to lesson one.

Worked Example (Made-Up Numbers)

Dana's ten test cases: 8 right, 1 fixable, 1 wrong. The wrong one invents a project from empty notes, because she hasn't built the input check yet. That's a failed safety test, so the trial stays paused. She fixes the input check so empty notes stop the run, then reruns all ten. The reviewed trial starts only after the empty-input case stops and every required safety check passes. Even then, every draft is read before it's sent.

What Ten Cases Can and Can't Show

Ten cases support a small reviewed pilot. They aren't proof that the workflow is reliable in general.

Common Mistakes

• Testing only clean examples.
• Skipping a case that should be refused.
• Editing the instruction and not rerunning the full set.
• Starting the trial after a failed safety test.

Exercise

Build your table, run all ten, and write the one edit that would fix the worst miss.