Checkpoint A · after lesson 12

Checkpoint A

Can you rebuild backpropagation from memory?

~40 min · 1 lab · builds on 12 A network that trains itself

This checkpoint builds on lessons 01–12. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

Twelve questions

Nothing new is taught here. Twelve questions from lessons 01 to 12 come in mixed order, and then you rebuild one piece of your engine from memory.

Answer each question from memory before you look anything up. Every reveal names the lesson to revisit if the answer surprised you.

$g = a \cdot b + a \cdot c + a$ with a = 2, b = 3 and c = −1. What is $\frac{\partial g}{\partial a}$ ?

Three paths send b = 3, c = −1 and 1, and they add: 3. Lesson 09.

A model's errors on four examples are 4, −4, 2 and −2. What is its mean squared error?

16 + 16 + 4 + 4 = 40, and 40 / 4 = 10. The signed errors average 0; the mean absolute error is 3. Lesson 02.

gradcheck_scalar returns False for a function built from your Values: at one input, backward and nudging differ by a relative error of 0.3. What does it most likely mean?

Nothing, as long as the absolute gap is below 1

The absolute gap depends on the size of the gradient; 30% of it is a bug at any size.

One of the operations in the function has a wrong local slope

A relative error of 0.3 is a 30% disagreement: far beyond rounding.

Nudging is unreliable, so the check can be ignored

A central difference is accurate to many digits away from kinks; it is the check to trust.

A local slope somewhere in the function does not match its measurement. Lesson 11.

Stack two straight lines. The first turns x into $4x - 2$ ; the second multiplies that result by −0.5 and adds 3. What is the slope of the result?

−0.5 · 4 = −2: $-0.5(4x - 2) + 3 = -2x + 4$ , still a straight line. Lesson 05.

a, b, c and d are Values, and you write L = (a * b + c) * d. Starting at L, how many _prev links do you follow to reach a?

3: from L to the sum, from the sum to the product, from the product to a. Each Value records only its direct parents. Lesson 07.

A program fits a line to 200 house sizes and prices and ends with w = 1450. What does 1450 tell you?

The programmer's estimate of the price per unit of size, typed in before training

Nobody typed it in; training chose it to fit the 200 houses.

Each extra unit of size adds about 1450 to the price, as far as these 200 houses show

w is the slope the data chose: price per unit of size.

It could be the slope or the starting price: a line reads the same either way

Swap them and the line changes: w = 1450 adds 1450 per unit of size, while b = 1450 would be the price at size 0.

The slope the data chose. Lesson 01.

Your backward walks the graph recursively. It passes every test, then fails on a model with 3,000 chained operations. Why?

A long chain's gradients shrink to nothing

Shrinking gradients give tiny numbers, but the walk still finishes. A recursive walk 3,000 calls deep breaks whatever the values are.

Each level of recursion is a frame on Python's stack, and Python stops at about 1,000

The walk goes as deep as the chain; an explicit stack in a list has no such limit.

Python's recursion limit. Lesson 10's walk keeps its own stack.

A knob sits at w = 0.5, its partial slope is −3, and lr = 0.05. Where is w after one step of gradient descent?

0.5 − 0.05 · (−3) = 0.65: against the slope. Lesson 03.

Your training loss swings and never settles, though every gradcheck passes and a smaller learning rate only delays the swings. What do you check first?

Nothing in the gradients: each backward starts from zero

backward adds to whatever .grad holds; only a new Value starts at zero.

Raise the learning rate, to get through the swings faster

Bigger steps overshoot further; the swings would grow.

That every .grad is reset to 0 each step, before backward

Without a reset, each step follows the sum of every gradient so far.

The reset. Lesson 12.

A model has 60,000 knobs and one forward pass takes 5 ms. How many seconds does one nudged gradient take, nudging each knob once?

60,001 passes (one where you stand, one per knob) · 5 ms = 300,005 ms, about 300 seconds, for one step. Lesson 06.

$f = (a \cdot b + c) \cdot d$ with a = 3, b = −2, c = 5 and d = 4. What is $\frac{\partial f}{\partial b}$ ?

d = 4 reaches the sum, passes through it unchanged, and the product sends b the other input, a = 3: 4 · 3 = 12. Lesson 08.

A neuron outputs a hard 0 or 1, and gradient descent leaves its knobs where they started. Which change lets it train?

Raise the learning rate: the step trains, only slowly

The measured slope is exactly 0, and any rate times 0 is 0.

Swap the step for sigmoid, whose output moves a little with every nudge

With a slope everywhere, every nudge moves the loss, so gradient descent has something to follow.

Keep the step and run more steps: it decides the same way sigmoid does

Same decisions, but no slope: steps of size 0 go nowhere, however many you take. Training needs the output to change a little when a knob does.

Swap in sigmoid. Lesson 04.

Rebuild backward

One function, from memory, in a scratch file that is not part of your ember, so your workspace stays as it is. The tests build graphs with the course's own Value and call your function, so it has to stand on its own.

What: in checkpoint_a.py, write def backward(root: Value) -> None without looking back. Put every Value that root depends on in a topological order, each after the Values it was made from, using your own stack instead of recursion. Then set root.grad to 1 and run each _backward from root back to the leaves.

Why: without it, backpropagation is something you followed once rather than something you can write. It is lessons 08 to 10 in one function, and lesson 14 rebuilds the same walk for tensors.

You rebuilt backpropagation from memory.

Two jobs: an order, then a loop. What must be true of the order, and what went wrong with breadth-first?
Depth-first with your own stack of (node, done) pairs: append a node once its parents are done. Then set root.grad = 1.0 and walk the order backward, calling each node's _backward.
Nearly the code: order, seen, stack = [], set(), [(root, False)]; ...; root.grad = 1.0; for node in reversed(order): node._backward().

backward(L) gives every gradient of L = (z - 3)^2 with z = w*x + b.
After backward(y), y.grad is 1.0.
x reaches c directly and through d; x's backward must wait for both shares (w.grad = 28).
A chain of 5,000 additions backpropagates without a RecursionError.
For y = exp(tanh(a) * b) the gradient of a is y * b * (1 - tanh(a)^2).

"""Checkpoint A: rebuild backpropagation from memory.""" from ember.scalar import Value def backward(root: Value) -> None: """Fill in .grad on every Value that root depends on, as Value.backward does: order the graph so every node comes after its inputs, seed root.grad with 1.0, and run each node's _backward from root back to the leaves. Use an explicit stack (these graphs can be 5,000 nodes deep) and do not call Value.backward.""" raise NotImplementedError("Rebuild backward(root) from memory.")Running code needs JavaScript. Above is checkpoint_a.py as this lab starts.

Polish a rough-cut gem

A lesson you finished with the solution loaded leaves a rough-cut gem. Each one is listed here, with a link to rebuild it.

No lesson needs polishing, or this browser has no progress stored yet.

Where this goes

Part II is done: an autograd engine on single numbers that trains a network. Part III, coming in Wave 2, rebuilds it for whole arrays of numbers at once.

Previous 12 A network that trains itself

Report a problem with this checkpoint