Assessments

Entry check and exit check

Two short checks. The entry check tells a participant, before Day 1, whether the prerequisites are in place. The exit check asks one or two questions per module objective, so a participant (or an instructor) can see which objectives landed. Neither is graded. Every question can be answered from the lectures and labs; none is a trick question.

How to use them

  • Entry check. Participants take it on the Before Day 1 page, the first step before Setup: about 15 minutes, on paper or in a notebook, with no web search. One rule applies: a participant who misses two or more of the three questions in an area does that area’s remediation, listed on the same page, before Day 1; one miss in an area needs no action. Send the page a week ahead, so there is time for the remediation.
  • Exit check. On Days 2 to 5, the 15 minutes of retrieval practice that open the day use five of these questions: about three from the previous day and two from earlier days. The picks and the routine are on each day page (Day 2, Day 3, Day 4, Day 5). Participants can also use the questions of the day’s modules as a self-check that evening, or all of them after Day 5. Answers are folded under each question. Each answer names the lab and exercise that produced the evidence, so a participant who misses a question knows which cell to rerun.
  • Numbers. Where an answer quotes a measured number, it is the build run recorded in data/baselines.json and quoted in the lecture. If your room ran on another runtime and got a different number, accept the room’s number with the same reasoning.
  • Labs whose real path has not run. The readiness page lists which labs have run on their real models, and where. A question on a lab that has not tests the reasoning of the lecture and the checkpoint, not a model result.

Entry check

Participants take this check on the Before Day 1 page, which also says how to read the score and what to study in each area.

Fifteen questions in five areas, three per area, matching the prerequisites on the home page. Allow about 15 minutes. Work on paper or in a notebook, without searching the web, and write each answer down before you open its folded key.

Python

E1. What does this expression evaluate to?

[x * x for x in range(5) if x % 2 == 0]
Answer

[0, 4, 16]. The comprehension keeps 0, 2 and 4 and squares them.

E2. What does this print?

class Counter:
    def __init__(self):
        self.n = 0

    def add(self, k=1):
        self.n += k
        return self

c = Counter().add().add(3)
print(c.n)
Answer

4. Each call to add changes the same object and returns it, so the calls chain: 0 + 1 + 3.

E3. A loop over rows read from a CSV file stops with this traceback. What is wrong, and what is the fix?

  File "count.py", line 7, in <module>
    total = total + row["count"]
TypeError: unsupported operand type(s) for +: 'int' and 'str'
Answer

row["count"] is a string (the CSV reader returns text), and Python will not add a string to an integer. Convert it: total = total + int(row["count"]). The last line of a traceback names the error; the line above it shows where it happened.

NumPy

E4. A has shape (3, 4) and b has shape (4,). What are the shapes of A + b and A @ b?

Answer

A + b is (3, 4): b is broadcast across the three rows. A @ b is (3,): a matrix-vector product.

E5. With x = np.arange(12).reshape(3, 4), what are x[1, :] and x[:, -1]?

Answer

x[1, :] is [4, 5, 6, 7] (the second row). x[:, -1] is [3, 7, 11] (the last column).

E6. X has shape (N, d). Write one line that divides each row by its Euclidean length, and say why keepdims=True matters.

Answer

X / np.linalg.norm(X, axis=1, keepdims=True). With keepdims=True the norms have shape (N, 1), which broadcasts across the d columns. Without it they have shape (N,), which NumPy tries to align with the columns and fails (or, if N == d, silently divides the wrong way).

Machine-learning basics

E7. Why are hyperparameters chosen on a validation split rather than on the test split?

Answer

Choosing on the test split makes the test score an optimistic estimate: the setting was picked because it did well on those examples. The test split is used once, after every choice is fixed. The workshop follows this rule in every lab.

E8. Logistic regression predicts p = \sigma(w^\top x + b) for a label y \in \{0, 1\}. Write the loss for one example, and its derivative with respect to the logit z = w^\top x + b.

Answer

The cross-entropy (log loss) \mathcal{L} = -\big[y \log p + (1 - y)\log(1 - p)\big]. Its derivative with respect to z is p - y: prediction minus target. The same form returns in Modules 1, 2 and 6.

E9. Gradient descent updates \theta \leftarrow \theta - \eta \nabla_\theta \mathcal{L}. What do you expect to see if the learning rate \eta is far too large?

Answer

The loss oscillates or grows instead of falling, because each step overshoots the minimum. It can also become nan.

PyTorch

E10. Put these five lines of one training step in order.

optimizer.step()
loss = F.cross_entropy(logits, y)
optimizer.zero_grad()
loss.backward()
logits = model(x)
Answer

optimizer.zero_grad(), logits = model(x), loss = F.cross_entropy(logits, y), loss.backward(), optimizer.step(). Putting zero_grad() right after step() instead is also correct; what matters is that gradients are cleared before the next backward(), because PyTorch adds to them.

E11. What does model.eval() change, and what does with torch.no_grad(): change?

Answer

model.eval() switches layers whose behavior differs between training and evaluation, such as dropout (turned off) and batch normalization (uses stored statistics). It does not stop gradients. torch.no_grad() stops autograd from recording operations, which saves memory and time when no gradient is needed. Several labs rely on the difference: Lab 5’s leak test runs in eval() mode.

E12. layer = nn.Linear(4, 3) is applied to a tensor of shape (8, 4). What is the output shape, and how many parameters does the layer have?

Answer

Output (8, 3). Parameters: a 3 \times 4 weight matrix and 3 biases, 12 + 3 = 15.

Mathematics

E13. Let y = \sigma(z) with z = wx. What is \dfrac{dy}{dw}?

Answer

By the chain rule, \dfrac{dy}{dw} = \sigma(z)\big(1 - \sigma(z)\big)\, x, using \sigma'(z) = \sigma(z)(1 - \sigma(z)).

E14. P(A) = 0.3 and P(B \mid A) = 0.5. What is P(A \text{ and } B)? And what is the expected value of one roll of a fair six-sided die?

Answer

P(A \text{ and } B) = P(B \mid A)\,P(A) = 0.15. The expected value is (1 + 2 + \dots + 6)/6 = 3.5.

E15. A model predicts the distribution (0.7, 0.2, 0.1) over three classes and the true class is the first. What is the cross-entropy, in nats? What is it for a uniform prediction over four classes? And what is the cosine similarity of the vectors (1, 0) and (1, 1)?

Answer

-\ln 0.7 \approx 0.357 nats. For a uniform prediction over four classes, \ln 4 \approx 1.386 nats: the starting loss of Lab 6’s four-class classifier. The cosine is 1 / \sqrt{2} \approx 0.707.

Exit check

Questions are numbered module.objective, with a letter when an objective has two. Objective numbers refer to the list at the top of each lecture page. Each question has its own link, with the dot replaced by a hyphen: #q10-2 for 10.2, #q12-3a for 12.3a. The day pages use these links. The outcome numbers refer to the workshop’s learning outcomes:

Outcome By the end of the workshop a participant can Modules
1 Explain the line of ideas from count-based language models to transformers, and say what problem each step solved 1–5
2 Implement the core of each model in PyTorch: skip-gram with negative sampling, an LSTM language model, seq2seq with attention, and a small GPT 2–5
3 Use the Hugging Face stack to tokenize, load, fine-tune and evaluate pretrained models, including LoRA 6, 7
4 Use the OpenAI and Claude APIs for prompting, structured output and tool use, and compare them on the same task 8
5 Explain RLHF end to end, train a toy version, and name its failure modes 9, 10
6 Measure calibration, explain how RLCD’s objective differs from RLHF’s, and use a calibrated decision model (Jev) through its API 11, 12
7 Build and evaluate a RAG pipeline with LlamaIndex and LangChain 13
8 Build a LangGraph agent with tools, state, human-in-the-loop checkpoints and confidence-gated control 14
9 Combine these into one system, evaluate it, and report accuracy, abstention and cost 15

Module 1 · Text as data

Objectives: lecture 1.

1.1 (Objective 1 · Outcome 1) At min_count = 2, part of the validation split maps to <unk>. Name one change that lowers that share, and one cost of making it.

Answer

Lower min_count to 1, enlarge the vocabulary another way, or tokenize at the character level. The cost: more vocabulary entries seen once or twice, whose counts are unreliable (Zipf’s long tail), or, for characters, longer sequences and perplexities that are no longer comparable with word-level ones. Evidence: Lab 1, Exercise 1 (coverage printed at min_count 1 and 2).

1.2a (Objective 2 · Outcome 1) What is the perplexity of a uniform model over a vocabulary of 65 characters, and why?

Answer
  1. Every token gets probability 1/65, so the mean negative log-probability is \ln 65 and its exponential is 65. Evidence: Lab 1, Checkpoint 3 (empty counts give perplexity equal to the vocabulary size).

1.2b (Objective 2 · Outcome 1) At the word level, the trigram model’s test perplexity (1,437) is higher than the bigram’s (304). Why does more context make it worse?

Answer

Sparsity. At n = 3, 77% of the test trigrams never occur in training, so most predictions fall back on small smoothed probabilities. More context helps only while the counts can support it. Evidence: Lab 1, Exercise 3 and the baseline card (word-level perplexities, labeled not comparable with later labs).

1.3 (Objective 3 · Outcome 1) Naive Bayes and logistic regression are both linear classifiers over word counts. How does each set its weights, and which one’s probabilities do you expect to be overconfident, and why?

Answer

Naive Bayes sets its weights by counting words per class, under the assumption that words are independent given the class. Logistic regression sets them by minimizing cross-entropy. Naive Bayes is the overconfident one: correlated words are counted as independent evidence, so the evidence is double-counted. The false assumption harms the probabilities more than the ranking. Evidence: Lab 1, Exercises 5 and 6 (both classifiers, per-class precision and recall, the most confident errors).

Module 2 · Word vectors and neural networks

Objectives: lecture 2.

2.1 (Objective 1 · Outcome 1) Why do hot and cold end up close together in a skip-gram embedding space?

Answer

The distributional hypothesis: words that occur in similar contexts get similar vectors. Antonyms share most of their contexts (“the water is ___“), so cosine similarity reflects shared contexts, not synonymy. Evidence: Lab 2, Exercise 2 (nearest neighbors of the probe words).

2.2a (Objective 2 · Outcome 2) Write the skip-gram negative-sampling loss for one center word w, its context word o and K noise words n_1, \dots, n_K. What is its value when all vectors are zero?

Answer

\mathcal{L} = -\log \sigma(u_o^\top e_w) - \sum_{k=1}^{K} \log \sigma(-u_{n_k}^\top e_w). With zero vectors every \sigma(0) = 1/2, so \mathcal{L} = (K + 1)\log 2: the first training loss in the lab. Evidence: Lab 2, Exercise 1 (Checkpoint 1 tests the all-zero case; the training cell asserts the first loss).

2.2b (Objective 2 · Outcome 1) What does negative sampling buy over the full softmax, and what does it give up?

Answer

Cost: each pair costs O(K) instead of O(|V|). It gives up normalized probabilities: it is a different objective, so it cannot be used to compute a perplexity. Evidence: lecture 2, section 4; Lab 2, Exercise 1.

2.3 (Objective 3 · Outcome 2) In Lab 2, averaged skip-gram embeddings with a feed-forward network scored 0.867 test accuracy, against 0.884 for TF-IDF with logistic regression on the same split. Give two reasons from the lecture why the dense model need not win.

Answer

Averaging discards word order and blurs the few distinctive words that decide a topic. The embeddings were trained on about one million tokens. On topic classification with plenty of labels, word identity carries most of the signal, which TF-IDF keeps. Whether dense features win is an empirical question. Evidence: Lab 2, Exercises 3 and 4 and the results table.

Module 3 · Sequence models

Objectives: lecture 3.

3.1 (Objective 1 · Outcome 2) Write the RNN step, and the two LSTM lines that update the cell state and the hidden state.

Answer

RNN: h_t = \tanh(W h_{t-1} + U x_t + b). LSTM: c_t = f_t \odot c_{t-1} + i_t \odot g_t and h_t = o_t \odot \tanh(c_t), with forget gate f_t, input gate i_t, candidate g_t and output gate o_t. Evidence: Lab 3, Exercises 1 and 4 (checked against nn.RNNCell and nn.LSTMCell).

3.2a (Objective 2 · Outcome 1) The RNN trained in Lab 3 has \lVert W \rVert \approx 5.2, yet its gradient still vanishes with distance. How?

Answer

The gradient across T - s steps is a product of one-step Jacobians, each with a factor \mathrm{diag}(1 - h_j^2). For saturated \tanh units that factor is near 0, so the product shrinks whatever \lVert W \rVert is. \lVert W \rVert > 1 only removes the guarantee of vanishing. Evidence: Lab 3, Exercise 3, part A (gradient norm against distance; the printed norm of W).

3.2b (Objective 2 · Outcome 1) Does gradient clipping fix vanishing gradients? What does the LSTM change?

Answer

No. Clipping caps the size of an update, which handles explosion. The LSTM adds a cell state with an additive, gated update; along that path the Jacobian is a product of forget gates, which the network can hold near 1. In the lab, 50 characters back, the LSTM’s gradient was about 5 \times 10^4 times the RNN’s. Evidence: Lab 3, Exercise 3, part B (clipping demonstration) and the LSTM gradient plot after Exercise 4.

3.3 (Objective 3 · Outcome 1) Why may Lab 3 compare the LSTM’s test perplexity (5.01) with Lab 1’s character 5-gram (6.25), but not with Lab 1’s word-level bigram (304)?

Answer

Perplexities are comparable only at the same token unit, on the same test text, vocabulary and scored positions. The LSTM and the 5-gram are both scored per character on the same 60,394 test characters with the same 65-character vocabulary. A word-level perplexity is on a different scale. Evidence: Lab 3, the results table (the checkpoint asserts the LSTM beats the trigram and the 5-gram).

Module 4 · Seq2seq and attention

Objectives: lecture 4.

4.1 (Objective 1 · Outcome 2) In the plain encoder-decoder, what is the decoder’s first state, and how is the training loss computed?

Answer

The last encoder state, s_0 = h_S. The decoder is trained with teacher forcing: its inputs are the gold target tokens shifted right by one, and the loss is the cross-entropy of each next target token, averaged over non-padding tokens. Evidence: Lab 4, Exercise 1 (forward_plain, seq2seq_loss).

4.2 (Objective 2 · Outcome 1) Without attention, Lab 4’s model gets about 0.9 exact match on inputs with one date and 0 on inputs with two or more, under the same training budget as the attention model. Explain why, and say why this is not the vanishing-gradient problem.

Answer

The whole source must pass through one fixed-size vector, while the information the decoder needs grows with the number of dates. That is a capacity limit: the fixed-vector bottleneck. Vanishing gradients are an optimization problem on long paths; the bottleneck would remain even with perfect training. Evidence: Lab 4, Exercises 2 and 3 (exact match by K, without and with attention).

4.3a (Objective 3 · Outcome 2) Write dot-product attention for a decoder state s_t over encoder states h_1, \dots, h_S with a padding mask. How does the additive score differ?

Answer

Scores e_{t,i} = s_t^\top h_i; set masked positions to -\infty; weights \alpha_{t,i} = \mathrm{softmax}_i(e_{t,i}) over source positions; context \bar{h}_t = \sum_i \alpha_{t,i} h_i. The additive score replaces the dot product with a small network, u^\top \tanh(W_q s_t + W_k h_i). Evidence: Lab 4, Exercises 3 and 4 (weights sum to 1; padded positions get exactly 0).

4.3b (Objective 3 · Outcome 1) A row of the attention heat-map puts 0.99 on one source character. What does that show, and what does it not show?

Answer

It shows which encoder state the decoder read at that step: a soft alignment. It does not show how the information was used, so it is evidence, not an explanation. The softmax is over source positions, never exactly 0 or 1. Evidence: Lab 4, Exercise 5 (heat-maps and the alignment hit rate).

Module 5 · The transformer

Objectives: lecture 5.

5.1 (Objective 1 · Outcome 2) Why are attention scores divided by \sqrt{d_k}? Use Lab 5’s measurement.

Answer

With unit-variance entries, q^\top k has variance d_k, so the score’s standard deviation grows as \sqrt{d_k}: the lab measured about 4, 8 and 16 for d_k = 16, 64 and 256, and about 1 after scaling. Large scores saturate the softmax, which becomes nearly one-hot with tiny gradients. Evidence: Lab 5, Exercise 1 and its provided measurement.

5.2a (Objective 2 · Outcome 2) With d_k = d / n_h, does going from one head to four change the number of parameters?

Answer

No. Four heads are four narrower lookups with the same total projection size. Evidence: lecture 5, section 4; the stretch of Lab 5 asserts the count does not change with n_h.

5.2b (Objective 2 · Outcome 2) List the parts of a pre-norm decoder-only transformer from token IDs to logits, and say what the causal mask guarantees.

Answer

Token embeddings plus position embeddings; n_\ell blocks, each x + \mathrm{Attn}(\mathrm{LN}(x)) followed by x + \mathrm{FFN}(\mathrm{LN}(x)); a final layer norm; the output layer. The mask sets scores of later positions to -\infty, so position t depends only on tokens 1, \dots, t, and one forward pass gives T valid predictions. Evidence: Lab 5, Exercises 0, 2 and 3 (the leak tests on one step and on the whole model).

5.3 (Objective 3 · Outcome 1) At full budget the mini-GPT scored 1.548 nats per character and the LSTM retrained with the same loop 1.573, from one seed; the seed-to-seed spread was 0.02 to 0.03 nats. What may you conclude?

Answer

A tie at this budget, not a ranking. The difference is within seed noise. The comparison is meaningful only with each model’s parameter count, steps and time stated; with a shorter budget the LSTM won clearly. Evidence: Lab 5, Exercise 4 (the results table).

Module 6 · Pretraining and the Hugging Face stack

Objectives: lecture 6.

6.1a (Objective 1 · Outcome 3) Byte-pair encoding on the corpus low ×5, lower ×2, newest ×6, widest ×3, with an end-of-word symbol: what is the count of the pair (e, s), and what is the first merge?

Answer

9 (6 from newest, 3 from widest). It ties with (s, t), also 9; ties go to the pair that sorts first, so the first merge is e + s. The lab’s first five merges are e+s, es+t, est+_, l+o, lo+w. Evidence: Lab 6, Exercise 1.

6.1b (Objective 1 · Outcome 3) In masked language modeling, what happens to the selected positions, and why does BERT have no perplexity?

Answer

15% of positions are selected; of those, 80% become [MASK], 10% a random token and 10% stay unchanged, and all selected positions are predicted. The model uses context on both sides, so it defines no left-to-right probability of a sentence: no perplexity in Module 1’s sense, and no generation. Evidence: Lab 6, Exercise 4 (the three shares).

6.2 (Objective 2 · Outcome 3) You load a pretrained body into AutoModelForSequenceClassification and see a warning about newly initialized weights. Is something broken? And what must always match the model?

Answer

Nothing is broken: the classification head is new and must be trained. The tokenizer must be the model’s own, because token IDs index rows of the model’s embedding matrix; the wrong tokenizer gives wrong rows and no error. Evidence: Lab 6, Exercises 2, 3 and 5 (AutoTokenizer and the AutoModelFor... classes).

6.3 (Objective 3 · Outcome 3) Before fine-tuning a four-class classifier, what validation loss do you expect, and which parameters does fine-tuning update?

Answer

About \ln 4 \approx 1.386: a fresh head gives near-uniform predictions. Standard fine-tuning updates every parameter, body and head, with a small learning rate for a few epochs; training only the head is a different, cheaper method. Evidence: Lab 6, Exercise 5 (the starting loss is checked against \log 4).

Module 7 · Fine-tuning and LoRA

Objectives: lecture 7.

7.1 (Objective 1 · Outcome 3) Instruction tuning changes which of these: the architecture, the loss, or the data? Which token positions are scored?

Answer

The data. The model and the next-token cross-entropy are unchanged; the examples are formatted with a chat template, and the loss is averaged over the response tokens only, including the closing <|im_end|>. Evidence: Lab 7, Exercises 3 and 4 (format_chat, build_labels).

7.2a (Objective 2 · Outcome 3) LoRA at rank r = 8 on q_proj (576 \to 576) and v_proj (576 \to 192) in each of 30 blocks: how many trainable parameters?

Answer

r(d_{\text{in}} + d_{\text{out}}) per layer: 8 \times 1{,}152 + 8 \times 768 = 15{,}360 per block, and 460{,}800 in all, about 0.34% of the model. Evidence: Lab 7, Exercise 5 (lora_param_count against peft).

7.2b (Objective 2 · Outcome 3) Why does LoRA training start exactly at the pretrained model, and why can it still move?

Answer

B = 0 at initialization, so BA = 0 and W = W_0. The gradient of B depends on Ax, which is not zero, so B moves at the first step; it is A’s gradient that is zero at first. After training, BA can be merged into W_0 at no inference cost. Evidence: Lab 7, Exercises 1 and 2 and the provided gradient cell.

7.3 (Objective 3 · Outcome 3) After fine-tuning, response perplexity falls and ROUGE-1 rises. Name one thing neither number shows, and two decoding settings you set when measuring.

Answer

Perplexity measures the probability of the reference responses, not the quality of generated text; ROUGE counts shared words with one reference, so a wrong answer can score well. Neither shows correctness: read the outputs. When measuring, decode greedily, and always set a stop token and a length limit. Evidence: Lab 7, Exercise 6 and the before-and-after checkpoint.

Module 8 · LLMs through APIs

Objectives: lecture 8.

8.1a (Objective 1 · Outcome 4) In a tool-calling loop, who runs the tool, and what goes back to the model?

Answer

Your code runs it; the model only writes a request. You append the model’s reply and one result per call, matched by the call’s ID. An unknown tool or bad arguments become an error result, not an exception. The loop stops at K_{\max} calls. Evidence: Lab 8, Exercise 4 (execute, run_tools).

8.1b (Objective 1 · Outcome 4) A structured-output call returns JSON that validates against your schema. What does that guarantee?

Answer

Form, not truth. It guarantees a parseable record; whether the fields are right is a separate number. Validity and exact match are reported separately. Evidence: Lab 8, Exercises 2 and 3 (extract with validate-and-retry; score).

8.2 (Objective 2 · Outcome 4) Why does Lab 8 count an invalid output as wrong, and print a standard error beside every exact-match rate?

Answer

If invalid outputs were dropped, a provider that fails more often could look more accurate: the denominator is always N. With N = 40 (or 12 on the CPU fallback), the standard error is large, so small differences between providers cannot be read. Evidence: Lab 8, Exercise 3.

8.3 (Objective 3 · Outcome 4) At $1 per million input tokens and $5 per million output tokens, what does a request with 400 input and 60 output tokens cost? Why does a tool loop’s input cost grow faster than its number of calls?

Answer

(400 \times 1 + 60 \times 5)/10^6 = \$0.0007. The API is stateless, so every call resends the whole message list, which grows by about \delta tokens per round: input tokens total K n_0 + \delta K(K-1)/2, quadratic in K. Evidence: Lab 8, Exercise 5 (cost_usd; measured input tokens against the formula).

Module 9 · Reinforcement and preference learning

Objectives: lecture 9.

9.1 (Objective 1 · Outcome 5) In the reinforcement-learning view of text generation, what are the state, the action and the policy?

Answer

The state is the prompt plus the response so far; the action is the next token; the policy is the language model’s next-token distribution. The log-probability of a response is the sum of its tokens’ log-probabilities. Evidence: Lab 9, Part A (the toy sequence task) and lecture 9, section 5.

9.2a (Objective 2 · Outcome 5) Write the REINFORCE loss with a baseline, and say why the baseline does not bias the gradient.

Answer

-\big((G - b).\mathrm{detach}() \cdot \log \pi_\theta\big), summed over positions and averaged over the batch. The score function \nabla \log \pi_\theta has mean zero, so subtracting a baseline that does not depend on the sampled response leaves the expected gradient unchanged and can cut its variance. Evidence: Lab 9, Exercises 1 and 2 (mean estimates against the exact gradient, with and without a baseline).

9.2b (Objective 2 · Outcome 5) Why must the leave-one-out baseline exclude the sample’s own reward, and how does the batch-mean baseline relate to it?

Answer

A baseline that includes the sample’s own reward depends on the response it is subtracted from, which shrinks the gradient. Using the batch mean as the baseline gives exactly (1 - 1/N) times the gradient that the leave-one-out baseline gives. Evidence: Lab 9, Exercise 2 (the exact identity is a checkpoint).

9.3 (Objective 3 · Outcome 5) Write the Bradley–Terry probability that response a is preferred to b, and the reward-model loss. Why can the reward model not reach 100% held-out accuracy?

Answer

P(a \succ b) = \sigma\big((g_a - g_b)/\tau_{\text{label}}\big); loss -\log \sigma\big(r_\phi(y_w) - r_\phi(y_l)\big). Raters are noisy and many pairs are close, so even the true rule would mislabel some pairs: the ceiling is \mathrm{Acc}^\star, often well below 1. Evidence: Lab 9, Exercises 3 and 4 (bt_prob, bt_loss, held-out accuracy beside \mathrm{Acc}^\star).

Module 10 · RLHF

Objectives: lecture 10.

10.1 (Objective 1 · Outcome 5) Name the three stages of the InstructGPT pipeline and what each produces.

Answer

Supervised fine-tuning on demonstrations (the SFT model, which becomes the reference); a reward model trained on pairwise comparisons; policy optimization against the reward model with a KL penalty toward the reference. Evidence: Lab 9 (the reward model) and Lab 10 (the policy step).

10.2 (Objective 2 · Outcome 5) Write the per-token reward Lab 10 uses. What does \beta control, and what does the KL penalty not protect against?

Answer

r_t = -\beta\big(\log \pi_\theta(y_t \mid s_t) - \log \pi_{\text{ref}}(y_t \mid s_t)\big) + \mathbf{1}[t = |y|]\, r_\phi(x, y). \beta sets the trade-off between reward and drift from the reference; changing it changes the optimum, not only the speed. The penalty punishes leaving the reference’s support, not concentrating on part of it, so it does not prevent mode collapse. Evidence: Lab 10, Exercises 2 and 3 (token_rewards, exact_kl).

10.3 (Objective 3 · Outcome 5) What is the DPO loss when the policy equals the reference, and what does DPO remove from RLHF?

Answer

\log 2: both implicit rewards are 0 and -\log \sigma(0) = \log 2. DPO removes the reward model and the sampling loop; it keeps the KL-regularized objective and the reference model. Both log-probabilities can fall during training; DPO raises the gap. Evidence: Lab 10, Exercise 5 (dpo_loss).

10.4 (Objective 4 · Outcome 5) With \beta = 0, the reward model’s score of the policy rises. What does the gold rule show, and why can the reward model not see it?

Answer

Reward hacking: the reward model’s score holds up while the gold reward ends lower than with the penalty, and the policy drifts far from the reference and loses variety. The gold rule lets the two scores part in three ways: no credit past three distinct positive words (the cap), a penalty for crowding, and simply fewer distinct positive words, or more negative ones, in the samples. The reward model was trained on the reference’s own samples. There the cap and the crowding term almost never fire, so it never learned them, and on text unlike those samples its score is extrapolation. Which route a run takes is not fixed: one exploratory run (seed 0, on an Apple M1 Pro laptop, not the T4 protocol) collapsed onto one positive word and filler, “good as as as …”, which neither the cap nor the crowding term penalizes. Other failure modes named in the lecture: mode collapse, sycophancy, and optimizing for what raters can judge rather than what is true. Evidence: Lab 10, Exercise 4 (the asserted signature; not yet verified across seeds on a T4).

Module 11 · Calibration

Objectives: lecture 11.

11.1 (Objective 1 · Outcome 6) A classifier gives confidence 0.8 on 1,000 cases. What must be true for it to be calibrated there? Can a calibrated model be useless?

Answer

About 800 of those cases must be right. Yes: a model that ignores its input and predicts the class frequencies (0.25 each on a balanced four-class set) is calibrated, with ECE 0, and is 25% accurate. Evidence: Lab 11, Exercise 1 (the input-blind check).

11.2 (Objective 2 · Outcome 6) Lab 1’s classifier at C = 1 has test accuracy 0.888 and mean confidence 0.723. Is it overconfident or underconfident? Why would you not read much into an ECE of 0.05 measured on 100 examples?

Answer

Underconfident: it is right more often than it claims (ECE 0.164). ECE has a positive noise floor: a perfectly calibrated forecaster measured on 100 examples with 15 bins scores about 0.07 on average. Evidence: Lab 11, Exercises 1 and 2 (the C sweep and the printed noise floor).

11.3 (Objective 3 · Outcome 6) Why is the Brier score a proper scoring rule while accuracy is not?

Answer

For a binary event with true probability p, the expected Brier score of a report q is (q - p)^2 + p(1 - p), minimized only at q = p: honesty pays. Expected accuracy is the same for every q > 0.5, so it cannot reward honest probabilities; the linear score even rewards overconfidence. Evidence: Lab 11, the stretch (expected scores on a grid), and Exercise 2.

11.4 (Objective 4 · Outcome 6) A wrong action costs 10 and deferring costs 1. Above what confidence should the system act? Does temperature scaling change the risk–coverage curve?

Answer

Act when \hat{p} > \lambda^* = 1 - 1/10 = 0.9. The curve depends only on the order of the confidences, which temperature scaling keeps for two classes (for four classes the top-class order changed on 0.92% of test pairs). What changes is where the cost-based threshold sits. Evidence: Lab 11, Exercises 3 and 5.

Module 12 · Calibrated decisions: RLCD and Jev

Objectives: lecture 12. These questions follow the honesty rule: they test the difference between what TypeSafe has stated and what is our illustration. The key for 12.3a follows lecture 12, section 5, as corrected from TypeSafe’s documentation on 2026-10-06; re-check it if the workshop lead’s sign-off changes that section.

12.1 (Objective 1 · Outcome 6) In the lecture’s framing (ours, not TypeSafe’s method), how does an outcome reward with a proper score differ from RLHF’s preference reward? Name one thing Lab 12’s toy experiment does not show.

Answer

A preference reward is relative, comes from a model of raters, and never sees a probability. An outcome reward is computed from a verified label and can score the whole reported distribution; with a proper score such as the Brier score, only the true probabilities earn the most reward, whereas an accuracy reward is maximized by putting all probability on one answer. The toy does not show how Jev was trained (TypeSafe has not published RLCD), nor that training for calibration beats repairing it afterwards: on held-out wordings both toy models were overconfident, and temperature scaling closed most of the gap. Evidence: Lab 12, Exercise 1 (the table on train, dev and test).

12.2 (Objective 2 · Outcome 6) With costs wrong = 20, ask = 0.5, miss = 4 and escalate = 3, compute \tau_{\text{act}} and \tau_{\text{esc}}. What does the system do at \hat{p} = 0.6?

Answer

\tau_{\text{act}} = 1 - 0.5/(20 - 4) = 0.96875 and \tau_{\text{esc}} = 1 - (3 - 0.5)/4 = 0.375. At 0.6 it asks: the expected costs are 8 (act), 2.1 (ask) and 3 (escalate). Evidence: Lab 12, Exercise 4 (the worked example is a checkpoint).

12.3a (Objective 3 · Outcome 6) Sort each statement into one of three groups: stated by TypeSafe in its own sources; reported by third parties and not found in a TypeSafe source; our illustration.

  1. “System One models are trained for calibrated decisions; validate their performance in the target domain.”
  2. RLCD optimizes calibrated probabilities instead of preference, as RLHF does.
  3. A Brier-score reward keeps a small decision model’s probabilities calibrated on its training items, while an accuracy reward drives them toward 1.
  4. Jev answers in 70 to 500 ms.
  5. A choice answer carries a probability for every option and a separate confidence that summarizes how concentrated the distribution is.
Answer

Stated by TypeSafe: 1 (typesafe-ai/skills, SKILL.md; the documentation’s System One page states the same first clause), 2 in substance, and 5 (typesafe-sdk, SKILL.md and the documentation’s Confidence page). For 2, TypeSafe’s AI primer (read 2026-10-06) says that RLCD “trains TypeSafe to return decisions and calibrated probabilities instead of generated text”, while RLHF “trains models to produce responses people prefer”; the words “instead of preference” are the third parties’ summary, not TypeSafe’s. Reported by third parties: 4. It is not in TypeSafe’s documentation, which says instead that “Most queries complete in about 100 ms”. Our illustration: 3, from Lab 12’s toy model; it is evidence about our toy, not about how Jev was trained. TypeSafe names RLCD and states its aim, but has not published its reward, data or algorithm. Evidence: lecture 12, section 5; Lab 12, Exercise 1 and the closing cell.

12.3b (Objective 3 · Outcome 6) A yes/no answer returns noul = 0.2. What is the chosen answer and its probability? A choice answer returns probabilities (0.6, 0.1, 0.1, 0.1, 0.1) and confidence = 0.5. Which number goes on the reliability diagram?

Answer

“No”, with probability 1 - 0.2 = 0.8 (“yes” when noul >= 0.5). For the choice answer, 0.6: the probability of the chosen answer. confidence measures how concentrated the distribution is; on calibrated synthetic data its ECE was above 0.10 while the probabilities’ was below 0.01. Evidence: Lab 12, Exercises 2 and 3 (chosen_answer, calibration_arrays).

Module 13 · Retrieval-augmented generation

Objectives: lecture 13.

13.1a (Objective 1 · Outcome 7) Name the seven stages of a RAG pipeline, in order.

Answer

Load, chunk, embed, index, retrieve, rerank, generate. Evidence: Lab 13, Exercises 2 to 5.

13.1b (Objective 1 · Outcome 7) What can a reranker not fix, and which number should a Jev reranker sort by?

Answer

It only reorders the k_0 candidates it is given, so it cannot recover evidence the retriever missed. Sort by the expected relevance level computed from the probabilities (score), not by confidence. Evidence: Lab 13, Exercise 4.

13.2a (Objective 2 · Outcome 7) Chunks of 512 tokens reach a higher recall@5 than chunks of 128. Why is that not enough to choose 512, and on which split do you choose?

Answer

Larger chunks contain more text, so they cover more evidence at the same k. Compare settings at a fixed context budget, n_{\text{ctx}} = kL. Choose on dev; report the choice on test once, with gained and lost questions. Evidence: Lab 13, Exercise 2 (the sweep at a fixed budget).

13.2b (Objective 2 · Outcome 7) Three questions have their first relevant chunk at ranks 3, 1 and nowhere in the list. What is the MRR?

Answer

(1/3 + 1 + 0)/3 \approx 0.444. Evidence: Lab 13, Exercise 1 (the lecture’s worked example is the checkpoint).

13.3 (Objective 3 · Outcome 7) An answer is fully faithful to its retrieved passages, and wrong. Where is it in the two-by-two table of evidence retrieved against correct, and what would you change first?

Answer

Evidence not retrieved, answer wrong: a retrieval failure. It is faithful to the wrong passages, so change retrieval first (chunking, k, hybrid retrieval, reranking). A faithfulness score is a judge model’s opinion until it has been compared with human labels. Evidence: Lab 13, Exercise 5 and the two-by-two table.

Module 14 · Agents

Objectives: lecture 14.

14.1 (Objective 1 · Outcome 8) What does an explicit LangGraph graph make visible that Module 8’s tool loop did not, and what stops a runaway loop?

Answer

The state, the nodes that update it, the edges and conditional routing that read it, and a checkpoint after every step. A runaway loop is stopped by your own count of model calls (K_{\max}, kept in the state) and a recursion_limit set on every run; the default in langgraph 1.2.12 is 10,007 steps. Evidence: Lab 14, Exercise 2 (a script that never finishes ends at K_{\max}).

14.2 (Objective 2 · Outcome 8) A node calls the decision model and then calls interrupt(). What happens when the run resumes, and how do you fix it? What does replay re-execute?

Answer

LangGraph reruns the interrupted node from its first line, so the decision model is called twice. Move the decision call into an earlier node; the lab checks it is called once. Replay re-executes every node after the snapshot, including tools that act on the world, so side effects must be idempotent. Memory comes from the thread: a second call with the same thread_id sees the earlier messages. Evidence: Lab 14, Exercises 3 and 4.

14.3a (Objective 3 · Outcome 8) The guard’s thresholds are \tau_{\text{esc}} = 0.375 and \tau_{\text{act}} = 0.969. What happens at p_{\text{allow}} = 0.6? Why does the guard compare noul itself rather than the probability of its chosen answer?

Answer

It asks: the run pauses for a person. The guard’s question is “is this email allowed?”, and the threshold is about the probability of “allowed”, so it compares noul directly; \max(\texttt{noul}, 1 - \texttt{noul}) would treat a confident “not allowed” as a reason to act. Evidence: Lab 14, Exercise 2 (region, after_guard).

14.3b (Objective 3 · Outcome 8) A guard scores an unsafe-action rate of 0 out of 45 disallowed items. What may you claim, and what else must you report?

Answer

Not that the guard is safe: 0 out of 45 is consistent with a true rate of several percent, and claiming below 1% would take hundreds of disallowed items with no failure. A guard that escalates everything also scores 0, so report the held-back rate and the number of interrupts beside it, with N. Evidence: Lab 14, Exercise 5 and the closing cell.

Module 15 · Capstone

Objectives: lecture 15. These questions were written from lecture 15 before 15-capstone.ipynb was built; check them against the notebook’s starter system before you use them. The capstone’s evaluation set does not exist yet, so no question here asks for a measured result.

15.1 (Objective 1 · Outcome 9) In the capstone starter, which nodes ask the decision model a question, and what does each decide?

Answer

plan asks whether the question needs evidence from two places; at p_{\text{multi}} \ge \tau_{\text{plan}} = 0.5 the generator splits it into two queries. verify asks whether every claim is supported; at p_{\text{sup}} \ge \tau_{\text{verify}} = 0.8 the draft is delivered, otherwise the system retrieves again with 2k chunks if a retry is left, and abstains if not. Evidence: Lab 15, the exercise (after_verify) and the self-test.

15.2a (Objective 2 · Outcome 9) With \ell_{\text{wrong}} = 5 and \ell_{\text{abs}} = 1, and 40 answerable and 25 unanswerable questions, what is the cost per question of abstaining on everything? When does answering pay?

Answer

40/65 \approx 0.62: each answerable question costs 1 when abstained on, each unanswerable one 0. Answering pays only when the answer is right more than 1 - 1/5 = 80\% of the time (Chow’s rule). Evidence: Lab 15, the scoring cell (lecture 15, section 3).

15.2b (Objective 2 · Outcome 9) A change gains 7 questions and loses 2 against the baseline. Is the improvement shown? What else do you compare g + l with?

Answer

Not at the 0.05 level: the sign test gives p \approx 0.18 (8 and 1 would give 0.039). Report it as “not shown”, not “no effect”. Compare g + l with the run-to-run flip count between two baseline runs; a change that moves fewer questions than the flips has shown nothing. Evidence: Lab 15, compare and flips on the share card.

15.3 (Objective 3 · Outcome 9) A two-minute share answers four questions. What are they, in order?

Answer

What you changed, and why the traces suggested it; what you predicted; what happened to the cost, accuracy and abstention, with g, l and p; what it cost in dollars or calls and in seconds. End with the one thing that surprised you. Evidence: Lab 15, the hypothesis card and the share card.