Syllabus

Forty-five lessons in seven parts, four checkpoints and a capstone. You start with a line that has two knobs and finish with a framework that trains convolutional and recurrent networks on a CPU or a GPU. Parts I and II are open now; the rest arrive in two more waves.

Each row gives the question a lesson answers. Every lesson runs on the code you wrote in the lessons before it; if you arrive in the middle, one button loads the finished pieces so you can carry on.

Nothing is locked. Your progress is stored only in this browser.

Progress needs JavaScript. Every released lesson reads without it.

1. Part I · Why learning works at all

   1. 01 A function with knobsCan a program find its own numbers?~30 minnot started
   2. 02 How wrong is wrong?Why square the error instead of adding it up?~35 minnot started
   3. 03 Downhill in the darkYou can only feel the slope underfoot. Now what?~40 minnot started
   4. 04 The simplest decisionWhat is the simplest thing that makes a decision?~40 minnot started
   5. 05 Bending spaceWhy stack layers, and why must there be a kink?~35 minnot started
   6. 06 The bill for nudging10,000 knobs: how many forward passes for one gradient?~30 minnot started

2. Part II · Autograd on single numbers

   1. 07 A number that remembersWhat must a number record so we can ask where it came from?~35 minnot started
   2. 08 Who owes the gradient?Each op knows only its local slope. Is that enough?~40 minnot started
   3. 09 Many paths, one sumA value used twice: why does = instead of += break training silently?~35 minnot started
   4. 10 The order of blameIn what order must the graph be visited?~40 minnot started
   5. 11 Trust, but verifyHow do you know your derivative is right?~45 minnot started
   6. 12 A network that trains itselfPut it all together: can your engine learn?~45 minnot started
   7. Checkpoint Anot started

3. Part III · Tensors and the backend seam Coming in Wave 2

   1. 13 Why arrays?Same math, far faster. Where did the time go?
   2. 14 The tensor engineCan the scalar engine's ideas carry over to tensors?
   3. 15 Adding a row to a whole tableHow does (64,10) + (10,) work?
   4. 16 Where did my gradient's shape go?The bias gradient came back (64,10). Why?
   5. 17 Matmul and its gradientCan you derive dA from shapes alone?
   6. 18 Reshape, index, reduceWhat is the gradient of picking, summing or reshaping?
   7. 19 The backend seamWhich array operations does your framework actually need?
   8. 20 Devices you can't mixWhat happens when data and weights live on different chips?

4. Part IV · A framework you'd want to use Coming in Wave 2

   1. 21 Where are all my weights?How does an optimizer find every weight in a nested model?
   2. 22 Why does a deep net go silent?Why do random weights of the wrong size kill a deep net?
   3. 23 What makes a good kink?What makes a good kink?
   4. 24 Scores into chancesWhy does exp overflow past about 88.7, and how do we stop it?
   5. 25 How surprised was the model?Why does a confidently wrong answer need a big gradient?
   6. 26 Stuck in the ravineCan a ball rolling downhill train faster?
   7. 27 Why not learn from all the data at once?Why learn from noisy minibatches instead of all the data?
   8. 28 Can it read handwriting?From ~88% with a hand-rolled loop to ~95% with a framework: what did the framework buy you?
   9. 29 Worse the longer it trainsWhy does the model get worse the longer it trains?
   10. 30 Save, load, inferHow do you keep what you learned?

5. Part V · Seeing: CNNs Coming in Wave 3

   1. 31 Convolution by handShift a digit two pixels: does your MLP still know it?
   2. 32 Convolution as a matmulCan we reuse the fast matmul?
   3. 33 Keeping training stableWhy does a deep CNN train so unevenly, and what fixes it?
   4. 34 A CNN that beats your MLPIs the idea worth the complexity?

6. Part VI · Remembering: sequences Coming in Wave 3

   1. 35 A word is not a numberHow do you feed a word to a matrix?
   2. 36 What came before?How can a network remember the previous character?
   3. 37 Exploding and vanishingWhy does the gradient 50 steps back come out as 10⁻¹² or 10¹²?
   4. 38 A memory that doesn't fadeHow do you build a memory that doesn't fade?
   5. 39 A character-level ShakespeareCan your framework write?

7. Part VII · Ready for transformers, ready for GPUs Coming in Wave 3

   1. 40 Deep stacks that still learnWhy can a 20-layer net still learn?
   2. 41 Attention, the operationHow can every position look at every other?
   3. 42 Start slow, end slowWhy start slow, and why end slow?
   4. 43 Where the memory goesWhy does training use 10× the memory of inference?
   5. 44 Where did my gradients go?Why do some gradients become zero in float16?
   6. 45 Out of the browserDoes your framework run on a real GPU unchanged?