Welcome. This deck is the motivation for the four days; the lecture notes in lectures/day1_llm_eval_foundations.md carry the detail. Ground rule for everything you will see today: every number on these slides was produced by the repository you cloned, in mock mode, and you can reproduce it with `make test-unit` or a notebook cell. There are no industry statistics, no named companies, no real incidents. Where we show a post-mortem it is a labelled composite built from the repo's own seeded weakness. Ask the room: who has an agent in production? Who has an eval suite that would catch a tool-selection regression? Keep the hands-up count for the last slide.

This is the diagnostic for vibe-driven development. If the answer is a feeling, you are shipping on vibes. It is not a criticism; it is where everyone starts. The point of the workshop is that the alternative is cheaper than it looks, because most of the eval suite is deterministic and runs offline.

Left column is not a strawman; it's how most of us started. Right column maps to: data/golden, src/stockroom/evals/metrics.py and judge.py, eval_thresholds.yaml with scripts/check_thresholds.py, src/stockroom/evals/otel_tracer.py, and the guards in src/stockroom/agent/harness.py. The transition is incremental: the deterministic metrics alone (day 1) already catch the failure on slide 10.

This is the thesis of the whole workshop. The state machine is Harness.run() in src/stockroom/agent/harness.py. Each guard is a class you can unit test. Make the point that "the model was fine" is a common and often true post-mortem conclusion: the model asked for a retry, the harness let it retry forever.

Say the label out loud. This is a teaching construct. The reason to do it as a post-mortem is that the shape is what you will recognise when it happens to you: a benign backend blip, a model that does the locally reasonable thing, a harness with one missing guard.

Walk the timeline. The important observation is at the bottom: an answer-only or tool-selection-only eval scores every one of these calls as correct. The defect is that nothing in the loop recognised "same call, same error, again". In the repo: StockroomTools.get_order_status() has an internal second attempt when the flag is off; with naive_retry on, the tool has no retry and the LegacyShardSimulator fails every attempt, and the HeuristicPlanner re-issues the call because that is what a model that sees "retry later" does.

Roughly five times the input tokens for zero useful output, and that is with MAX_STEPS=8. Ask the room what their harness's step limit is. Many do not have one; then the number is bounded by the model's context window and your patience. The token numbers are the fake client's chars/4 estimate, not a tokenizer; the ratio is what matters.

Do not let anyone leave with a dollar figure in their head as "the cost of the incident". The point is the shape: a ×4–5 per-request cost for zero value, with no ceiling except MAX_STEPS. With a bigger model, a bigger context, or no step limit, the multiplier grows. The Nova Pro numbers are the ones scripts/fetch_pricing.py pulled from the public Price List API on 2026-10-06; Claude prices are not in that feed and are null in the file.

This slide is the argument for evaluating the harness. The model-side fix (a better prompt saying "don't retry more than twice") is the weakest of the four because it depends on the model obeying it every time.

Three layers, one flag or case each. Day 3 is where participants switch each flag on and watch which metric moves. Today just establish that each layer has its own failure shape and its own metric.

This is the model being wrong and the harness making it recoverable. The metric that sees it is invalid_call_rate; the golden case allows the recovery (in_order_subset, max_steps 3).

test_ambiguous_tool_description_drops_tool_selection() asserts exactly this: answer_correctness > tool_selection_accuracy. This is the single most convincing slide for "evaluate trajectories, not answers". Nobody changed the model. Someone edited a docstring. The other two environment flags: oversized_payload takes G049 from 3.4k input tokens to 14k and a TOKEN_BUDGET termination; injection_unguarded makes the agent follow a maintenance note in restock_policy.md section 4 and create a 10,000-unit request.

Rebut with the table two slides back. Mention that the golden set records expected tools and arguments per case precisely so this can be measured without a model.

Kappa 0.25 is barely above chance even though "58%" sounds half-right: v1 passes everything. The two remaining v2 disagreements are items that are string-correct but misleading, which no fact matcher catches; that's what a real cross-family judge is for in live mode. Attribution for the practice comes on Day 2: Hamel Husain's "Your AI Product Needs Evals" and Shankar et al., "Who Validates the Validators?".

tests/test_trajectory_regression.py writes the run-to-run spread and case intervals; scripts/check_thresholds.py checks the evidence and applies eval_thresholds.yaml. Day 4 shows how to set the allowed regression from the nightly spread.

The flywheel is the course structure. Each day adds one stage and connects it to the previous one. Emphasise that the loop closes: a production failure seen in a trace becomes a golden case, which becomes a gate.

Point out the offline-first rule: no credentials needed for any of the four labs. Live mode needs model access, two model IDs, and optionally an S3 bucket and a Knowledge Base; the README lists the prerequisites and the cost expectations.

Return to the hands-up count from the first slide. The commitment: every participant leaves able to answer "better or worse, by how much" for their own agent, using this repo as a template.

Transition to the lab. First cell is detect_mode(); if anyone sees LIVE, they have credentials and model IDs in their environment and should decide deliberately whether they want to spend.