Measuring Time Saved and Quality

If you don't measure, you'll keep an agent that doesn't help and drop one that does. Measure two things: time saved and quality.

The people, companies and numbers in this lesson are made up. Check any figure against your own situation before you rely on it.

Get a Baseline

Time ten real runs of the task by hand and write down each time in minutes. Use the average. A guess from memory is usually too low or too high.

A Results Table (Maya, Made-Up Numbers, Three Runs Per Week)

• Week | Runs this week | Avg minutes per run before | Avg minutes per run now (with review) | Outputs checked | Outputs with errors | Tool cost this week (dollars)
• 1 | 3 | 30 | 14 | 3 | 1 | 5
• 2 | 3 | 30 | 10 | 3 | 1 | 5
• 3 | 3 | 30 | 8 | 3 | 0 | 5
• 4 | 3 | 30 | 8 | 3 | 0 | 5

How To Read It

Week four: 22 minutes saved per run, 3 runs, so 66 minutes saved that week, or 1.1 hours. Maya has 12 clients a month, which is about 3 a week, so that is 4.4 hours a month, the same figure as the cost example. Tool cost of 5 dollars a week is about 20 dollars a month, also the same. With only 3 runs a week, check every output. Quality is outputs without errors divided by outputs checked: 2 of 3 in week one, 3 of 3 in week four.

A Decision Rule

• Keep: time saved is at least a third and the error rate is falling or under your limit.
• Fix: time is saved but errors are above your limit. Change the instruction and rerun the test table.
• Stop: you spend as long reviewing as you saved, or one error is costly and repeats.

Common Mistakes

• Counting only the model's time. Fix: count your review and fix time too.
• Checking only good-looking outputs. Fix: check all of them while volume is low, and sample at random once it grows.
• Changing the task and the agent at once. Fix: change one thing and rerun.

Exercise

• Measure your baseline on ten runs, then set a target such as halving the time with no drop in quality.
• Fill one row of the results table each week for four weeks.
• Pass condition: a baseline, a target, four weekly rows with units, and a keep, fix or stop decision after row four.