Learning objectives. By the end of this notebook you can:
Instrument an agent run with OpenTelemetry (GenAI semantic conventions), read the span tree and the gen_ai.* attributes, and ship the same spans to Arize Phoenix.
Detect loops, repeated calls and context growth from a run or from its finished trace with ToolCallEvaluator.
Do error analysis before picking metrics: read raw outputs, write down what went wrong, then turn the notes into checks.
Calibrate an LLM judge against human labels (agreement, Cohen’s kappa, confusion matrix) and probe it for position, verbosity and self-preference bias.
Change a rubric and measure whether the judge got better, not just different.
Three graded exercises; each check cell prints not solved yet until your code passes, and the Exercise checklist cell near the end lists their status (STOCKROOM_STRICT_EXERCISES=1 makes an unsolved exercise an error). Everything runs offline in mock mode with a deterministic fake model and fake judge.
Setup
import importlib.utilimport osimport subprocessfrom pathlib import Path# Published repository (docs/DECISIONS.md entry 2). Only used when the package is not already# installed, e.g. in a fresh Colab or SageMaker Studio kernel.REPO_URL ="https://github.com/genial-labs-ai/system_agent_harness_aws"REPO_DIR = Path("stockroom-workshop")os.environ.setdefault("DEEPEVAL_TELEMETRY_OPT_OUT", "1")if importlib.util.find_spec("stockroom") isNone:ifnot REPO_DIR.exists(): subprocess.run(["git", "clone", "--depth", "1", REPO_URL, str(REPO_DIR)], check=True)# Pins mirror pyproject.toml; everything else (boto3, pydantic, OpenTelemetry, ...) is# resolved from the package metadata by the editable install.%pip install -q "deepeval==4.2.8""ragas==0.4.3""mcp==2.3.0""jsonschema==4.26.0""pyyaml==6.0.3""langchain-community>=0.3.27,<0.4""pandas>=2.2""matplotlib>=3.9"-e ./stockroom-workshopprint("stockroom importable:", importlib.util.find_spec("stockroom") isnotNone)
stockroom importable: True
from stockroom.config import detect_modeconfig = detect_mode()
/home/runner/work/system_agent_harness_aws/system_agent_harness_aws/.venv/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
1. Error analysis before metrics
Before choosing metrics, look at raw outputs and write down what is actually wrong. This is the practice described by Hamel Husain in Your AI Product Needs Evals (https://hamel.dev/blog/posts/evals/) and studied by Shankar et al. in Who Validates the Validators? (https://arxiv.org/abs/2404.12272): people only discover what their grading criteria should be by reading outputs, and the criteria keep drifting as they read more.
We will do a small version of that. Run eight golden queries with the ambiguous_tool_desc weakness on, read the trajectories, and then decide what to measure.
I found 1 matching product: SKU-1019 Ear Defenders 31dB ($19.90, 33 in stock).
1
G007
How many units of SKU-1015 are in stock?
[search_products]
I found 5 matching products: SKU-1015 Nitrile Gloves Large (box of 100) ($12.90, 42 in...
2
G011
How much stock is left for SKU-1001?
[search_products]
I found 5 matching products: SKU-1001 A4 Copy Paper 80gsm (500 sheets) ($6.90, 320 in ...
3
G014
What's the status of order ORD-1001?
[search_products]
I found 1 matching product: SKU-1001 A4 Copy Paper 80gsm (500 sheets) ($6.90, 320 in s...
4
G017
Give me the tracking number for ORD-1008.
[search_products]
I found 1 matching product: SKU-1008 Cordless Drill Driver 18V ($129.00, 5 in stock).
5
G026
What is the return window for unused items?
[search_policy_docs]
From Stockroom Returns Policy — Return window: Unused items in their original packagin...
6
G033
Is SKU-1008 at or below its reorder point? If so, raise an urgent restock request for ...
[search_products]
I found 5 matching products: SKU-1008 Cordless Drill Driver 18V ($129.00, 5 in stock);...
7
G037
Write me a short poem about warehouses.
[]
That's outside what I can help with. I can answer questions about Stockroom products, ...
Reading the table, the stock and order questions were answered from search_products results. The answers often contain the right number, so an answer-only grader would pass them — yet the agent is doing the wrong thing and will break the moment a product has several variants. The observation becomes a metric: tool selection accuracy, which evaluate_case already computes. Keep a notes table like this one for every eval you build; the metrics come second.
scores = pd.DataFrame( [ evaluate_case(by_id[c], weak.run(by_id[c].query, case_id=c), judge).model_dump()for c in sample_ids ])[["case_id", "tool_selection", "answer_correctness_deterministic", "judge_passed"]]scores
case_id
tool_selection
answer_correctness_deterministic
judge_passed
0
G001
1.0
True
True
1
G007
0.0
True
True
2
G011
0.0
True
True
3
G014
0.0
False
False
4
G017
0.0
False
False
5
G026
1.0
True
True
6
G033
0.0
False
False
7
G037
1.0
True
True
2. Instrumenting runs with OpenTelemetry
configure_tracing(TraceExporter.MEMORY) returns a TracingHandle with an in-memory exporter; RunTracer(handle) is what the harness uses to open one invoke_agent span per run, one chat {model} span per model call and one execute_tool {tool} span per tool call. Attribute names follow the OpenTelemetry GenAI semantic conventions (status Development; the names are spelled out in stockroom/evals/semconv.py and cross-checked against the installed opentelemetry-semantic-conventions package by tests/test_tracer.py).
One model-call span and one tool span, attribute by attribute:
def gen_ai_attributes(span) ->dict[str, object]:return {k: v for k, v insorted(span.attributes.items()) if k.startswith("gen_ai.")}chat_span =next(s for s in spans if s.name.startswith("chat "))tool_span =next(s for s in spans if s.name =="execute_tool get_stock_level")print("chat span:", chat_span.name)for k, v in gen_ai_attributes(chat_span).items():print(f" {k} = {v!r}")print("\ntool span:", tool_span.name)for k, v in gen_ai_attributes(tool_span).items():print(f" {k} = {v!r}")print("\nworkshop-specific attributes on the tool span:")for k, v insorted(tool_span.attributes.items()):if k.startswith("stockroom."):print(f" {k} = {v!r}")
chat span: chat fake.stockroom-planner-v1
gen_ai.operation.name = 'chat'
gen_ai.provider.name = 'stockroom.fake'
gen_ai.request.max_tokens = 1024
gen_ai.request.model = 'fake.stockroom-planner-v1'
gen_ai.response.finish_reasons = ('tool_use',)
gen_ai.response.model = 'fake.stockroom-planner-v1'
gen_ai.usage.input_tokens = 856
gen_ai.usage.output_tokens = 4
tool span: execute_tool get_stock_level
gen_ai.operation.name = 'execute_tool'
gen_ai.provider.name = 'stockroom.fake'
gen_ai.tool.call.arguments = '{"sku": "SKU-1008"}'
gen_ai.tool.call.id = 'tooluse_a6cbfc828455'
gen_ai.tool.call.result = '{"sku": "SKU-1008", "name": "Cordless Drill Driver 18V", "stock_level": 5, "reorder_point": 10, "below_reorder_point": true, "location": {"aisle": 2, "bin": 2}, "unit": "each", "supplier_name": "Ironbridge Tools", "lead_time_days": 5}'
gen_ai.tool.description = "Get the stock level (units in stock), reorder point and warehouse location (aisle, bin) for one product by SKU. Use for 'how many', 'how much', 'left', 'check', 'where is it stored' questions about a SKU."
gen_ai.tool.name = 'get_stock_level'
gen_ai.tool.type = 'function'
workshop-specific attributes on the tool span:
stockroom.step = 1
stockroom.tool.is_error = False
stockroom.tool.payload_chars = 234
stockroom.trajectory.depth = 1
Optional: the same spans in Arize Phoenix
configure_tracing(TraceExporter.PHOENIX) swaps the in-memory exporter for OTLP/HTTP. Start a local Phoenix with make phoenix (needs uv sync --extra phoenix) and set PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006; the cell is skipped otherwise so that the notebook never waits on a collector that is not there.
phoenix_endpoint = os.environ.get("PHOENIX_COLLECTOR_ENDPOINT")if phoenix_endpoint: phoenix_handle = configure_tracing(TraceExporter.PHOENIX, endpoint=phoenix_endpoint) Harness(config, tracer=RunTracer(phoenix_handle)).run(case.query, case_id=case.id) phoenix_handle.flush()print(f"exported one run to Phoenix at {phoenix_endpoint}; open it in the browser")else: installed = importlib.util.find_spec("phoenix") isnotNoneprint("Phoenix export skipped: PHOENIX_COLLECTOR_ENDPOINT is not set "f"(arize-phoenix installed: {installed}; run `make phoenix` and export the variable)." )
Phoenix export skipped: PHOENIX_COLLECTOR_ENDPOINT is not set (arize-phoenix installed: False; run `make phoenix` and export the variable).
3. Loops and context growth with ToolCallEvaluator
ToolCallEvaluator works from a RunResult (from_run) or from finished spans (from_spans), so the same checks can run in CI and over traces collected in production. Two seeded weaknesses:
naive_retry: the legacy order shard is “down” and the model re-issues the identical call;
oversized_payload: search_products dumps the whole catalogue into the context.
data/judge_calibration/calibration_v1.jsonl holds 32 (query, answer, context) triples labelled pass/fail by the authors, including paraphrases, hallucinated extras, partial answers and two “string-correct but misleading” items. Eight of them are bias probes and are excluded from the agreement numbers.
FakeJudge has two rubric versions on purpose: v1 passes any answer containing at least half of the key facts (lenient, verbosity-blind), v2 requires every key fact and no forbidden content. calibrate() grades every item and compares with the human label.
from stockroom.evals.calibration import calibrate, run_all_probesfrom stockroom.evals.judge import load_rubricitems = load_calibration()print(f"{len(items)} items, {sum(1for i in items if i.probe_group)} of them bias probes\n")print(load_rubric("answer_correctness", "v1"))print("\n---\n")print(load_rubric("answer_correctness", "v2"))
32 items, 8 of them bias probes
# Rubric: answer correctness (v1)
**Task.** Decide whether the assistant's answer to the user's question is correct, using the
reference answer as ground truth.
**Pass** if the answer conveys the main point of the reference answer.
**Fail** if the answer is off-topic or contradicts the reference answer.
Score from 0 to 10: 10 = fully correct, 5 = partially correct, 0 = wrong.
Return JSON: {"score": <0-10>, "pass": <true|false>, "rationale": "<one sentence>"}
---
# Rubric: answer correctness (v2)
**Task.** Decide whether the assistant's answer to the user's question is correct and complete,
using the reference answer (and, when given, the tool/context output) as ground truth.
Evaluate in this order:
1. **Key facts.** List the key facts in the reference answer: quantities, prices, dates, ids
(SKU-####, ORD-####, RSR-####), statuses, policy numbers (days, percentages, thresholds).
Every key fact must appear in the answer. Equivalent forms count ("thirty" = 30, "1,250" =
1250, "$8.50" = "8.50 dollars").
2. **Contradictions.** The answer must not state anything that contradicts the reference or the
context, and must not add unsupported claims presented as fact (invented dates, extensions,
reservations, free services).
3. **Scope.** If the user asked for several things, all of them must be answered.
4. **Verbosity is not quality.** Do not reward length, politeness or formatting. A short answer
containing every key fact is a 10.
Scoring (0–10): 10 = all key facts, no contradictions; 7–9 = all key facts but minor unsupported
detail; 4–6 = some key facts missing; 0–3 = wrong, contradictory, or refuses when it should answer.
**Pass** requires a score of 7 or more.
Return JSON: {"score": <0-10>, "pass": <true|false>, "rationale": "<one sentence citing the facts checked>"}
fake-judge rubric v1: n=24 agreement=58.33% kappa=0.250
judge pass judge fail
human pass 10 0
human fail 10 4
fake-judge rubric v2: n=24 agreement=91.67% kappa=0.833
judge pass judge fail
human pass 10 0
human fail 2 12
v1 disagreements:
item
human_pass
judge_pass
judge_rationale
0
C09
False
True
v1: 1/1 key facts present
1
C10
False
True
v1: 2/2 key facts present
2
C11
False
True
v1: 1/1 key facts present
3
C12
False
True
v1: 2/2 key facts present
4
C13
False
True
v1: 1/2 key facts present
5
C14
False
True
v1: 1/2 key facts present
6
C15
False
True
v1: 1/2 key facts present
7
C16
False
True
v1: 1/2 key facts present
8
C23
False
True
v1: 1/1 key facts present
9
C24
False
True
v1: 1/1 key facts present
v2 disagreements:
item
human_pass
judge_pass
judge_rationale
0
C23
False
True
v2: 1/1 key facts present
1
C24
False
True
v2: 1/1 key facts present
Agreement went up and the false-positive count went down, but v2 still disagrees with the humans on C23 and C24: answers whose key facts are all present yet which add invented projections. A fact-matching rubric cannot see that; this is where a real model judge (live mode) earns its cost.
Bias probes
Three probes reuse the same answers under different presentation: swapped order (position), padded wording (verbosity) and an “authored by your own model family” label (self-preference). A fair judge scores the pairs identically.
probes = pd.DataFrame( [ {"probe": p.probe, "group": p.group, "detail": p.detail, "biased": p.biased}for p in run_all_probes(FakeJudge("v2"), items) ])probes
probe
group
detail
biased
0
position
position:P1
preferred answer scored 1.00 when first and 1.00 when second
False
1
verbosity
verbosity:V1
short=1.00 verbose=1.00 (same facts)
False
2
self_preference
self_pref:S1
same_family=1.00 other_family=1.00
False
Exercise 1 — rubric v3: catch the two misleading answers
Write a JudgeV3 that keeps everything v2 does and additionally fails answers that present speculation as fact (C23 projects usage “for months”; C24 claims the warranty is “usually” extended). Implement it as a subclass of FakeJudge that post-processes the v2 verdict.
Success criteria: agreement and kappa are higher than v2’s on the calibration set, and the three bias probes still report no bias.
from stockroom.evals.judge import JudgeVerdictclass JudgeV3(FakeJudge):"""Rubric v3 = rubric v2 + a check for speculative language.""" name ="fake-judge"def__init__(self) ->None:super().__init__("v2")self.rubric_version ="v3"def grade(self, item: JudgeInput) -> JudgeVerdict: verdict =super().grade(item)# TODO: if the answer contains speculative phrasing, return a failing copy of `verdict`# (verdict.model_copy(update={...})) with a rationale that says why.return verdict
report_v3 = calibrate(JudgeV3(), items)v3_probes = run_all_probes(JudgeV3(), items)if report_v3.agreement == report_v2.agreement and report_v3.kappa == report_v2.kappa: exercise_pending("day2.ex1", "v3 behaves exactly like v2")else:print(report_v3.summary())assert report_v3.agreement > report_v2.agreement, "agreement did not improve over v2"assert report_v3.kappa > report_v2.kappa, "kappa did not improve over v2"assertnotany(p.biased for p in v3_probes), "v3 introduced a bias" exercise_passed("day2.ex1",f"agreement {report_v2.agreement:.2%} -> {report_v3.agreement:.2%}, "f"kappa {report_v2.kappa:.3f} -> {report_v3.kappa:.3f}, disagreements "f"{[d[0] for d in report_v3.disagreements]}", checked=report_v3, )
Exercise 1: not solved yet (v3 behaves exactly like v2)
Exercise 2 — add a disagreement to the calibration set
A calibration set is only useful if it contains items the judge gets wrong. Write a new CalibrationItem (a query, an answer, the context the answer was produced from, the human label) on which FakeJudge("v2")disagrees with your human label. Hint: v2 only checks facts and forbidden strings.
Success criteria: the item validates, its id is new, and the v2 verdict differs from human_label.
from stockroom.evals.calibration import to_judge_inputfrom stockroom.evals.golden import CalibrationItemprint(items[22].model_dump_json(indent=2)) # C23, for the shape
{
"id": "C23",
"query": "How many SKU-1015 are in stock?",
"answer": "SKU-1015 has 42 units in stock, which means 42 boxes of 100 so roughly 4,200 pairs are ready to ship; no reorder needed for months.",
"context": "Tool result get_stock_level: SKU-1015 Nitrile Gloves Large (box of 100), stock_level 42, reorder_point 10, aisle 3 bin 2.",
"expected_facts": [
"42"
],
"forbidden_facts": [
"0 units"
],
"human_label": "fail",
"human_score": 2,
"rationale": "The usage projection is unsupported speculation presented as fact.",
"probe_group": null,
"probe_role": null
}
def new_calibration_item() -> CalibrationItem |None:"""A labelled item on which rubric v2 disagrees with the human (None = not solved yet)."""# TODO: return CalibrationItem(id="C33", query=..., answer=..., context=..., expected_facts=[...],# forbidden_facts=[...], human_label="pass" | "fail", human_score=1..5, rationale=...)returnNone
new_item = new_calibration_item()if new_item isNone: exercise_pending("day2.ex2")else:assertisinstance(new_item, CalibrationItem)assert new_item.idnotin {i.idfor i in items}, "pick an unused id" v2_verdict = FakeJudge("v2").grade(to_judge_input(new_item))assert v2_verdict.passed != new_item.human_pass, (f"v2 agrees with you (judge pass={v2_verdict.passed}); make the item harder" ) exercise_passed("day2.ex2",f"human={new_item.human_label} judge v2 pass={v2_verdict.passed} ({v2_verdict.rationale})", checked=new_item, )
Exercise 2: not solved yet
Exercise 3 — find the loop in a trace from its spans
ToolCallEvaluator.from_spans did this for you in section 3. Now do it by hand from the execute_tool spans of the naive_retry run (still in handle): group tool spans by (gen_ai.tool.name, gen_ai.tool.call.arguments) and return the most repeated signature.
Success criteria: most_repeated_tool_call(spans) returns (tool_name, count) matching the loop finding that ToolCallEvaluator reported for the same run.
from stockroom.evals import semconv as scretry_spans = [ s for s in handle.finished_spans() if s.attributes.get(sc.GEN_AI_OPERATION_NAME) =="execute_tool"]print(f"{len(retry_spans)} tool spans; attribute keys to use: {sc.GEN_AI_TOOL_NAME!r}, "f"{sc.GEN_AI_TOOL_CALL_ARGUMENTS!r}")
8 tool spans; attribute keys to use: 'gen_ai.tool.name', 'gen_ai.tool.call.arguments'
def most_repeated_tool_call(tool_spans) ->tuple[str, int] |None:"""(tool name, number of identical calls) for the most repeated call (None = not solved)."""# TODO: count (gen_ai.tool.name, gen_ai.tool.call.arguments) pairs and return the top one.returnNone
answer = most_repeated_tool_call(retry_spans)if answer isNone: exercise_pending("day2.ex3")else: loop =next(f for f in retry_report.findings if f.kind =="loop") expected = (loop.signature.split(":", 1)[0], loop.count)assert answer == expected, f"got {answer}, ToolCallEvaluator says {expected}" exercise_passed("day2.ex3", f"{answer[0]} was called {answer[1]} times with identical arguments")
Exercise 3: not solved yet
Save your work for Day 4
The Day 4 gate’s answer_correctness is graded by a judge, so the capstone review should say how far that judge can be trusted. This cell saves what you measured today: agreement and kappa for rubric v2 and for your rubric (Exercise 1), and your calibration item (Exercise 2), which Day 4 hands to the gate’s judge. It writes reports/participant/day2.json (or into STOCKROOM_HANDOFF_DIR) once both exercises have passed, saving what the checks passed: after changing your rubric or item, re-run its check first. Otherwise Day 4 uses the reference artefact from data/handoff/ and says so.
from stockroom.exercises import PASSED, exercise_status, still_checkedfrom stockroom.handoff import CalibrationSummary, Day2Handoff, save_handoffitem_now = new_calibration_item()ifnotall(exercise_status(e) == PASSED for e in ("day2.ex1", "day2.ex2")):print("Nothing saved: Exercises 1 and 2 have not both passed, so Day 4 will use the reference ""calibration.")elifnot (still_checked("day2.ex1", report_v3) and still_checked("day2.ex2", item_now)):print("Nothing saved: your rubric or calibration item changed after its check passed. Re-run ""the Exercise 1 and 2 checks, then this cell.")else: own_calibration = Day2Handoff( mode=config.mode, calibrations=[CalibrationSummary.from_report(r) for r in (report_v2, report_v3)], calibration_item=item_now, ) saved = save_handoff(own_calibration)print(f"saved rubrics {[c.rubric_version for c in own_calibration.calibrations]} and item "f"{own_calibration.calibration_item.id} to {saved}")
Nothing saved: Exercises 1 and 2 have not both passed, so Day 4 will use the reference calibration.
Exercise checklist
One line per graded exercise. With STOCKROOM_STRICT_EXERCISES=1 this cell fails unless every exercise passed; make notebooks runs the solution notebooks that way.
Read outputs first, write notes, then turn the notes into metrics (Hamel Husain’s and Shankar et al.’s advice); tool_selection came out of exactly that loop today.
One invoke_agent span per run with chat and execute_tool children is enough to recompute loop and context-growth metrics after the fact, in Phoenix or in CloudWatch.
A judge is a model with its own failure modes: measure agreement and kappa against human labels, keep disagreements in the calibration set, and re-run the bias probes after every rubric change.
Day 3 opens the harness itself: the state machine, guards, compaction and the MCP tool boundary.