From Traditional NLP to Modern LLMs

Welcome: the setup check and the week ahead

Genial Labs · The Day 1 opening

One line of ideas

Modern language models did not appear from nowhere.

  • Each module fixes a specific failure of the one before it: from counting words to attention, to training on human preferences, to agents.
  • You build each step yourself, in a notebook, and see it work.
  • The same small datasets come back all week, so every improvement is measured against the module before it, not asserted.

The five days

Day Theme Modules
Day 1 Foundations: from counts to attention 1 Text as data, 2 Word vectors and neural networks, 3 Sequence models, 4 Seq2seq and attention
Day 2 Transformers and pretrained models 5 The transformer, 6 Pretraining and the Hugging Face stack, 7 Fine-tuning and LoRA
Day 3 Using and aligning LLMs 8 LLMs through APIs, 9 Reinforcement and preference learning, 10 RLHF
Day 4 Calibration, decisions and retrieval 11 Calibration, 12 Calibrated decisions: RLCD and Jev, 13 Retrieval-augmented generation
Day 5 Agents and the capstone 14 Agents, 15 Capstone

Module 0 is optional pre-work on your own laptop. Nothing later depends on it.

Each day answers one question

  1. Day 1. How do we turn text into something a model can learn from?
  2. Day 2. How does one architecture, pretrained at scale, become a general tool?
  3. Day 3. How do we use these models, and what are they trained to want?
  4. Day 4. When should a model’s answer be trusted, and how do we ground it in sources?
  5. Day 5. How do we build reliable systems out of these models?

How a module works

On Day 1, each module is 95 minutes: 45 minutes of lecture, then 50 of lab. On Days 2 to 5, each module is 120 minutes: 55 minutes of lecture, 55 of lab, then a 10-minute debrief. On Days 2 to 4 the second module breaks for lunch between its lecture and its lab. On Day 5, the capstone build runs on across lunch. The capstone (Module 15) has no lecture: it takes 11:30–12:25, 13:25–14:30 and 14:45–16:45 on Day 5.

Every lab follows one rhythm:

  1. Predict what a cell will output before you run it. Write the prediction down.
  2. Run it.
  3. Explain the result, especially when it differs from your prediction.
  4. Check your work against a checkpoint that passes or fails.

The timetable

Day 1 · 95-minute modules

Time
08:00–09:00 Drop-in clinic: Module 0 pre-work (optional)
09:00–09:10 Welcome, setup check
09:10–10:45 1 · Text as data: lecture 45 · lab 50
10:45–11:00 Break
11:00–12:35 2 · Word vectors and neural networks: lecture 45 · lab 50
12:35–13:35 Lunch
13:35–15:10 3 · Sequence models: lecture 45 · lab 50
15:10–15:25 Break
15:25–17:00 4 · Seq2seq and attention: lecture 45 · lab 50

Days 2–5 · 120-minute modules

Time
09:00–09:15 Retrieval practice
09:15–11:15 Lecture 55 · Lab 55 · Debrief 10
11:15–11:30 Break
11:30–12:25 Lecture 55 (Day 5: hands-on)
12:25–13:25 Lunch
13:25–14:30 Lab 55 · Debrief 10 (Day 5: hands-on)
14:30–14:45 Break
14:45–16:45 Lecture 55 · Lab 55 · Debrief 10 (Day 5: hands-on)
16:45–17:00 Synthesis (Day 5: Synthesis and wrap-up)

Breaks and lunch keep their length. From Day 2, each day opens with retrieval practice and closes with a synthesis of the day’s results. The full schedule is on the site.

Setup: nothing to install

  • Every lab runs in Google Colab, in the browser. You need a Google account.
  • When a lab asks for a GPU: Runtime → Change runtime type → T4 GPU. The free tier is enough.
  • API keys are optional. With none, the API labs run on a small open model and you still complete every exercise.
  • If you have keys, put them in Colab Secrets, never in a cell: OPENAI_API_KEY, ANTHROPIC_API_KEY, TYPESAFE_API_KEY.
  • Details are on the Setup page.

Setup check: do this now

  1. Open 00-setup.ipynb in Colab.
  2. Choose Runtime → Run all.
  3. It prints four blocks: the runtime, the keys, the provider, and the checkpoint.
  4. The last line should read Checkpoint passed. You are ready for Day 1.

Tell us the provider line it printed: open, a keyed provider, or an error. If a cell stops with an error, raise a hand now, not in Lab 1.

The same data all week

Data Used in
arXiv Topics v1 Modules 1, 2, 6 and 11
Tiny Shakespeare Modules 1, 3 and 5
Workshop Desk Decisions v1 Modules 11, 12 and 14
Workshop Lectures v1 Modules 13, 14 and 15

Lab 1 prints two numbers, an n-gram perplexity and a classifier accuracy. Every later module is measured against them.

Working in the labs

  • Each exercise is a TODO stub with its solution folded directly beneath it. Try first; open the solution when you are stuck, run it, and go on.
  • A checkpoint after every exercise says whether it checked your code or the reference.
  • The core path fits the lab slot. The stretch section at the end is optional: fast groups may start it, the room as a whole never does.
  • In the debrief, your numbers go on the board. Keep them.

Where everything is

Next: Module 1 · Text as data