This deck carries the Day 4 teaching touchpoints: the 09:00 recap, the 90-minute lecture block (09:15 to 10:45), the lab briefing, the 16:00 review and the workshop close-out. The lecture notes hold the detail, including everything quoted from AWS documentation; participants read them beforehand. Every number on these slides comes from this repository in mock mode; each note says how to reproduce it. The five "nightly" values on the threshold slides are the lecture's illustrative, synthetic teaching data and are labelled as such. Before the session: start a full make ci in a spare terminal so a fresh run is ready for the afternoon (instructor guide timing note).

Timetable from docs/INSTRUCTOR_GUIDE.md section 1. Suggested split of the 90 minutes: 10 min recap and motivating failure; 10 min offline, nightly, online; 25 min the three numbers with the prediction checkpoint; 15 min what aggregates hide and the gate summary; 10 min labels from the data; 10 min red team and the CI order; 10 min AWS map and the lab briefing. The AWS material is the part most likely to be out of date: it summarises vendor pages fetched on 2026-10-06. Send people to lecture section 6 and the links there rather than quoting it from a slide.

09:00 recap. Ask people to open the renamed reports/eval_results.json files from Day 3; the capstone compares against them. Reproduce: STOCKROOM_WEAKNESSES=<flag> make eval, then read "metrics" in reports/eval_results.json, or the Day 3 notebook's per-flag charts. Mean input tokens are chars/4 estimates, rounded.

Every aggregate stays above its floor and within 0.05 of the baseline: two broken cases out of 50 move a rate by 0.04. Reproduce: the "gate and its blind spot" section of the Day 4 notebook runs scripts/check_thresholds.py's evaluate() with the aggregate-only thresholds (eval_thresholds.yaml without category_gates and cost) on each configuration. The 15% is 2019.8 → 2323.3 mean input tokens against reports/baseline/main.json. Do not reveal yet that oversized_payload passes the same gate; the capstone has participants find it.

Exercise numbering follows notebooks/src/day4_bedrock_evaluations_and_ci_gating.py, whose introduction marks the required offline path and the optional AWS sections. If it is renumbered, follow the exercise checklist cell at the end.

Lecture section 1. The nightly workflow keeps its filename but has no cron: it runs only when someone dispatches it, so Bedrock spend is always a deliberate act (docs/DECISIONS.md). The flaky-eval table (lecture section 3) is pre-reading; the one convention to say out loud: tests fail only on hard invariants, the gate script fails on soft metrics, so one regressed case produces one readable table instead of a wall of red.

Lecture section 2.3. Why 2·√2·sd: a PR run and the baseline run each carry noise sd, so their difference has sd·√2; two of those is the margin below which a drop is plausibly noise. Mock mode first: run the suite three times and every per-repeat value is 1.0 with sd 0 (the Day 4 notebook's first threshold cell; STOCKROOM_EVAL_REPEATS=3 make eval writes the same run_to_run block to reports/eval_results.json). That is reproducibility, not reliability: it is why the PR gate can use the floors directly, and why the numbers that matter for thresholds come from live runs.

Two to three minutes; let people compute the standard deviation on paper or in a terminal. Collect yes/no answers for 1 and 2 by show of hands before the reveal. The values are the lecture's and the notebook's ILLUSTRATIVE_NIGHTLY numbers; they exist so the procedure can be taught before anyone has a live run.

Reproduce the statistics: uv run python -c "from stockroom.evals.stats import floor_is_safe, required_max_drop, run_to_run; r = run_to_run([0.86, 0.90, 0.84, 0.88, 0.87]); print(round(r['mean'], 3), round(r['sd'], 4), floor_is_safe(r['mean'], r['sd'], 0.85), required_max_drop(r['sd']))" It prints 0.87 0.0224 False 0.07. The pass/fail table is what the real gate (evaluate() in scripts/check_thresholds.py) returns in the notebook's regression-decisions cell; Exercise 3 asserts the same verdicts. For the tool-selection runs (0.90, 0.88, 0.92, 0.89, 0.91) the same command prints 0.9 0.0158 True 0.05. When a results file carries run_to_run with more than one repeat, noise_notes() prints these warnings in the summary on its own.

Lecture section 2.4. Reproduce the right-hand column with uv run pytest tests/test_seeded_weaknesses.py -k pr_gate (test_pr_gate_rejects_every_seeded_weakness() asserts the shipped gate rejects each flag and passes the fixed agent), and the left-hand column with the notebook's verdicts(AGGREGATE_ONLY) cell. The category rule is exact in mock mode, where every category terminates as its cases expect; in a live gate a two-case category moves in steps of 0.5, which is a discussion question for the review.

Excerpt of reports/summary.md from that run (also posted as the PR comment in CI). A faster reproduction than the full make ci: STOCKROOM_WEAKNESSES=naive_retry make eval promptfoo, then make thresholds. The paired list compares each case with its own outcome in the committed baseline (reports/baseline/main.json, written by make baseline). The gate refuses to compute deltas against a baseline whose mode, dataset hash, judge or agent model differ (baseline_problems()). Unset the variable and the same command passes.

Exercise 1 of the Day 4 notebook. The check cell expects exactly S007, S010, S011 and S012 rejected, S013 accepted although the agent fails it. Reproduce with the solutions notebook (notebooks/solutions/Day4_Bedrock_Evaluations_and_CI_Gating.ipynb), whose validation table prints the reasons and the agent's result side by side. S007 is a near-duplicate of G019 by trajectory signature: same category, same expected call. It is a good prompt for "what is a duplicate?". In live mode the teacher is a Bedrock model that paraphrases the templates; the labels still come from the data files.

Lecture section 4. Reproduce: the in-process red-team cells of the Day 4 notebook (same test definitions, same assertion functions as make promptfoo), or STOCKROOM_WEAKNESSES=injection_unguarded make promptfoo. The full Promptfoo suite is 18 tests: 10 golden-slice cases and 8 red-team cases. Watch the mechanism, not the pass mark: a red-team case that passes with quarantined 0 deserves a second look. Either retrieval did not surface the injected chunk, or the model ignored it by luck.

Lecture section 5; make ci runs the same steps in the same order as .github/workflows/agent_eval_ci.yml. The OIDC trust policy and the least-privilege permissions policy are in docs/aws/ and lecture section 7 (pre-read). Demo if time allows: STOCKROOM_WEAKNESSES=ambiguous_tool_desc make ci, whose summary shows tool_selection_accuracy 0.640 against min 0.85 and baseline 1.000 (delta -0.360).

Lecture section 6 is pre-reading and the source for every statement here (AWS pages fetched on 2026-10-06; verify before quoting). The limitation that shapes the workshop: none of the Bedrock Evaluations job types evaluates an agent trajectory, so trajectory metrics live in the repo or in AgentCore Evaluations. The notebook's job builder writes one JSONL line per golden case (50 lines) and validates the CreateEvaluationJob request against botocore's service model with no network call. The dataset limits are in lecture section 6.2.

Matches the lab plan in lecture section 9. Facilitation notes: docs/INSTRUCTOR_GUIDE.md section 4 (Day 4). For the capstone, accept any rule the evidence supports (per-category termination, a token-cost limit, a new golden case), and insist that the review says what happens to the committed baseline. Run the failing make ci with the flag exported in that shell only.

The capstone's check cell writes the before/after gate summaries for naive_retry, the thresholds and the review under reports/day4/; those are what the review block compares. Compare the participants' rules with what the repository ships: eval_thresholds.yaml version 2 carries category_gates and a cost limit, and test_pr_gate_rejects_every_seeded_weakness() holds the property.

Questions 1, 2, 3, 4 and 7 of lecture section 9. Spend most of the hour on capstone reviews: put two participants' before/after summaries side by side and ask the group which review they would approve. Common mistakes to surface (lecture section 11): tuning thresholds to the last run; keeping only the synthetic cases the agent passes; trusting aggregates over small categories; committing a baseline from a run with a flag on; red-team assertions on text only.

Close-out for the workshop. Return to the hands-up count from the Day 1 opening deck (who had an eval suite that would catch a tool-selection regression). Ask each participant for one thing they will add to their own agent's pipeline next week, and which file in this repository they will copy it from. Collect lab feedback through the repository's issue forms (.github/ISSUE_TEMPLATE). Remind everyone that live AWS spend stays opt-in: nothing in the repo creates resources or submits jobs without explicit consent.