Lesson 12 · Part II · Autograd on single numbers

A network that trains itself

Put it all together: can your engine learn?

~45 min · 3 labs · builds on 11 Trust, but verify

This lesson builds on lessons 01–11. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

Two gradient checks: backward 1.00 against nudging 1.02, and backward 100 against nudging 102. Which is more suspicious?

The second: its gap is 2, the first's only 0.02

The gap has to be read against the size; the relative errors are equal.

Neither: they are equally suspicious, both off by about 2% of their size

0.02 / 1.02 and 2 / 102 are both about 0.0196.

They are equally suspicious, at a relative error of about 0.0196 each, and both would fail a tolerance of 0.00001.

One weight w multiplies the input of each of three examples, and the loss adds the three products: $L = w \cdot 3 + w \cdot 5 + w \cdot (-2)$ . After one backward from L, what does w.grad hold?

3: the share from the first use, because a Value gets its gradient from one place

Every use of w moves L when w moves, so every use sends a share.

2: the average of the three shares

backward only adds. If you want an average, the loss has to divide by 3 itself.

6: each use sends back its own input, and the shares add up, 3 + 5 − 2

w is one Value used three times, so three shares arrive and backward adds them.

6. Nudge w by 0.01 and L moves by 0.06: every use of a shared Value pays in, and the shares add.

Put it together

Your Value engine computes exact gradients for any expression of +, *, powers, exp, log, tanh and relu, and your gradcheck proves each one.

That is everything a neural network needs: neurons that weigh their inputs, layers of them, a loss, and a loop that steps downhill. Can your engine learn?

The line of lesson 03, w and b, is fitted with your Values: build the loss, call loss.backward(), step each knob. One version resets w.grad and b.grad to 0 before every backward; the other never resets. What does the version that never resets do?

The same as the other: every backward starts from zero

backward adds to what .grad already holds, so without a reset the old gradients stay in.

Its loss swings up and down and never settles: each step follows the sum of every gradient so far

The old gradients keep pushing after the knobs have passed the bottom, so the walk overshoots again and again.

It trains faster: more gradient means bigger steps, and bigger steps get there sooner

Bigger steps overshoot, and here they also point the way the loss was sloping many steps ago.

It swings and never settles. Without a reset, a knob's .grad is the sum of every gradient since training began, so each step also pushes along old directions. The cell below fits the line both ways. With a reset the loss falls from 41.00 to 0.02 by step 10. Without one it is 44.01 at step 10, reaches 1.50 at step 40 and is back up at 40.10 by step 59.

The same line fitted twice with your Values, with and without resetting the grads.

from ember.scalar import Value

points = [(1.0, 3.0), (2.0, 5.0), (3.0, 7.0), (4.0, 9.0)]


def train(reset, steps=60, lr=0.02):
    w, b = Value(0.0), Value(0.0)
    losses = []
    for _ in range(steps):
        loss = Value(0.0)
        for x, y in points:
            loss = loss + (w * x + b - y) ** 2
        loss = loss * 0.25
        if reset:
            w.grad = 0.0
            b.grad = 0.0
        loss.backward()
        w.data -= lr * w.grad
        b.data -= lr * b.grad
        losses.append(loss.data)
    return losses


for reset in [True, False]:
    losses = train(reset)
    shown = ", ".join(f"{losses[i]:.2f}" for i in [0, 10, 20, 30, 40, 50, 59])
    print(f"reset before backward: {str(reset):5}  loss at steps 0, 10, ..., 59: {shown}")

reset before backward: True   loss at steps 0, 10, ..., 59: 41.00, 0.02, 0.01, 0.01, 0.01, 0.01, 0.01
reset before backward: False  loss at steps 0, 10, ..., 59: 41.00, 44.01, 32.61, 14.41, 1.50, 2.47, 40.10

Neurons, layers, a network

A neuron owns one weight per input and a bias, all of them Values. It outputs tanh of its weighted sum plus its bias. A layer is a list of neurons that all read the same inputs. An MLP, a multi-layer perceptron, feeds each layer's outputs to the next layer, and its last layer skips the tanh so that it can output any number. parameters() collects every weight and bias, and zero_grad() sets every grad back to 0 before a backward.

A neuron with weights 0.5 and −1 and bias 0.1, reading inputs 2 and 1, has the weighted sum 0.5 · 2 + (−1) · 1 + 0.1 = 0.1, and outputs tanh(0.1) ≈ 0.0997.

An MLP reads 2 inputs, has two hidden layers of 4 neurons, and one output neuron. Every neuron has a weight per input and a bias. How many parameters does it have?

First layer: 4 neurons · (2 weights + 1 bias) = 12. Second: 4 · (4 + 1) = 20. Output: 1 · (4 + 1) = 5. Total 37.

What: write Neuron. It keeps one weight per input, each a Value starting at a random number between −1 and 1, and a bias Value starting at 0. Calling it returns tanh of the weighted sum plus the bias, or the plain sum for an output neuron. parameters() lists the weights, then the bias. Also write zero_grad(), shared by every part of the network, which sets every parameter's grad to 0.

Why: without Values for weights, backward can't reach them and nothing trains. The next lab builds layers from these neurons, and the training loop calls zero_grad() before every backward; the cell that never reset shows why.

Your ember has a neuron that can learn.

Lesson 04's neuron took its weights as plain numbers. What must they be now, so that backward can reach them?
In __init__, keep one Value per input in self.w, each starting at a random number between -1 and 1 (random.uniform draws one), keep the bias in self.b as a Value starting at 0.0, and remember nonlin. In the call, start a running total at the bias and add each weight times its input; squash the total with its tanh() method when nonlin is True. zero_grad walks self.parameters() and sets each grad to 0.0.
Nearly the code: total = self.b, then a for loop over zip(self.w, x) that does total = total + w * xi, then return total.tanh() when self.nonlin, else total. For parameters(), copy the weights into a new list with list(self.w), append self.b, and return the list.

Neuron(3) has three weight Values in [-1, 1] and a bias Value that starts at 0.
Two neurons built one after the other start with different weights.
With w = [0.5, -1, 0.25] and b = 0.1, the output on [1, 2, 3] is tanh(0.5 - 2 + 0.75 + 0.1).
Neuron(2, nonlin=False) returns w . x + b, which may be larger than 1.
parameters() lists the weight Values in order, then the bias: the same objects.
After backward, a weight's gradient is its input times 1 - tanh^2.
zero_grad() sets the grad of every parameter back to 0.0.

"""Neurons, layers and a multi-layer perceptron built from Values (lesson 12).""" import random from collections.abc import Sequence from .scalar import Value class _Module: """What every piece of a scalar network shares: resetting the grads of the Values its own parameters() lists. Each subclass defines parameters(), which returns the list of Values that training adjusts. """ def zero_grad(self) -> None: """Set the grad of every parameter back to 0.0, ready for the next backward.""" # --- gap L12.zero_grad --- raise NotImplementedError("Set the grad of every Value in self.parameters() back to 0.0.") # --- end --- class Neuron(_Module): """tanh(w . x + b), or w . x + b when nonlin is False. Weights start uniform in [-1, 1] (from Python's random module, one draw per input, in order) and the bias starts at 0. Attributes: w: The weights, a list of Values, one per input. b: The bias, a Value. nonlin: True when the output is squashed with tanh. """ def __init__(self, in_features: int, nonlin: bool = True) -> None: """Make a neuron that reads in_features inputs. Args: in_features: How many inputs the neuron reads, and so how many weights it has. nonlin: True to squash the output with tanh, False to output w . x + b as it is. """ # --- gap L12.neuron_init --- raise NotImplementedError("Keep in_features random weights in [-1, 1] as Values in the list self.w, a bias Value(0.0) in self.b, and nonlin in self.nonlin.") # --- end --- def __call__(self, x: Sequence[Value | float]) -> Value: """The neuron's output on the inputs x. Args: x: The inputs, plain numbers or Values, one per weight. Returns: A Value: between -1 and 1 when nonlin is True, else any number. """ # --- gap L12.neuron_call --- raise NotImplementedError("Add up w * x over the inputs, starting from the bias; squash with tanh when self.nonlin.") # --- end --- def parameters(self) -> list[Value]: """The Values that training adjusts. Returns: A new list: the weights in input order, then the bias. It holds the neuron's own Values, not copies. """ # --- gap L12.neuron_parameters --- raise NotImplementedError("Return the weights followed by the bias, as one list.") # --- end --- class Layer(_Module): """out_features neurons that all read the same inputs. Calling it returns a list of Values. Attributes: neurons: The layer's Neurons, a list of out_features of them. """ def __init__(self, in_features: int, out_features: int, nonlin: bool = True) -> None: """Make a layer of out_features neurons, each reading in_features inputs. Args: in_features: How many inputs every neuron reads. out_features: How many neurons the layer has, and so how many outputs it returns. nonlin: Handed to every neuron: True to squash each output with tanh. """ # --- gap L12.layer_init --- raise NotImplementedError("Keep out_features Neurons, each reading in_features inputs, in the list self.neurons.") # --- end --- def __call__(self, x: Sequence[Value | float]) -> list[Value]: """Every neuron's output on the same inputs x. Args: x: The inputs, plain numbers or Values, in_features of them. Returns: A list of out_features Values: item i is neuron i's output. """ # --- gap L12.layer_call --- raise NotImplementedError("Return a list with each neuron's output on x.") # --- end --- def parameters(self) -> list[Value]: """The Values that training adjusts. Returns: A new list: every neuron's parameters(), neuron by neuron. It holds the layer's own Values, not copies. """ # --- gap L12.layer_parameters --- raise NotImplementedError("Return every neuron's parameters, neuron by neuron, as one list.") # --- end --- class MLP(_Module): """Layers of the sizes in layer_sizes after in_features inputs, tanh on every layer except the last. Calling it returns one Value when the last layer has one neuron, else a list. Attributes: layers: The network's Layers, a list in order from the inputs to the output. """ def __init__(self, in_features: int, layer_sizes: Sequence[int]) -> None: """Make the network's layers. Args: in_features: How many inputs the network reads. layer_sizes: How many neurons each layer has, in order; at least one layer. The last size is the number of outputs. """ # --- gap L12.mlp_init --- raise NotImplementedError("Keep one Layer per size in layer_sizes in the list self.layers; only the last layer skips tanh.") # --- end --- def __call__(self, x: Sequence[Value | float]) -> Value | list[Value]: """The network's output on the inputs x. Args: x: The inputs, plain numbers or Values, in_features of them. Returns: One Value when the last layer has one neuron, else a list with one Value per neuron of the last layer. """ # --- gap L12.mlp_call --- raise NotImplementedError("Feed x through every layer; unwrap a single output.") # --- end --- def parameters(self) -> list[Value]: """The Values that training adjusts. Returns: A new list: every layer's parameters(), layer by layer from the inputs to the output. It holds the network's own Values, not copies. """ # --- gap L12.mlp_parameters --- raise NotImplementedError("Return every layer's parameters, layer by layer, as one list.") # --- end ---Running code needs JavaScript. Above is ember/scalar_nn.py as this lab starts.

What: write Layer, neurons that read the same inputs and return a list of outputs, and MLP, which chains layers (tanh on every layer but the last) and returns a single Value when the last layer has one neuron; both list their parameters() in order.

Why: without layers, a network is one neuron drawing one straight cut. The next lab trains an MLP, and the decision boundary on the course's front page came from one. Part IV rebuilds the same pieces for tensors as ember.nn.Module and Linear.

Your ember has neural networks.

A layer is several neurons reading the same inputs; a network is layers where each reads the previous layer's outputs. How many inputs does each layer's neurons have?
Layer: make out_features Neurons, each reading in_features inputs and given the Layer's nonlin, and let the call return a list with one output per neuron. MLP: pair each layer size with the size before it, starting from in_features, and make one Layer per pair; only the last Layer gets nonlin=False. The call passes x through the layers in turn and unwraps a list of one.
Nearly the code: widths = [in_features, *layer_sizes]; one Layer(n_in, n_out, nonlin=k < len(layer_sizes) - 1) for each k, (n_in, n_out) in enumerate(zip(widths, widths[1:])). For parameters(), start from an empty list and extend it with each layer's parameters() in order; Layer does the same over its neurons.

Layer(3, 2) holds two different Neuron objects, each with three weights.
Calling a layer returns a list with each neuron's output.
Layer(3, 2).parameters() is neuron 0's four Values, then neuron 1's.
MLP(2, [4, 4, 1]) has layers of 4, 4 and 1 neurons; only the last is linear.
MLP(2, [3, 1]) returns one Value, not a list.
MLP(2, [3, 2]) returns a list of two Values.
MLP(2, [4, 4, 1]) has 12 + 20 + 5 = 37 parameters, first layer first.
With a hidden tanh of 1 and an output weight of 10, the output is 10 * tanh(1) = 7.6.
A first-layer weight's gradient matches a nudge of that weight.
model.zero_grad() resets all 37 parameters.

A forward pass through 10,000 Values takes far longer than the same arithmetic on plain floats. Where does the time go?

Into backward: autograd is the slow part

The forward pass with Values is already slow before backward starts, and backward costs about as much again.

Into making and linking a Python object for every single number, forward and backward alike

Each Value is an object with its own closure; creating, linking and visiting 10,000 of them costs far more than the multiplications.

Into the arithmetic: products and sums are expensive

The plain-float version does the same arithmetic in a small fraction of the time.

The per-number overhead. Every number is a Python object, and that, not the arithmetic and not backward itself, is where the time goes. The cell below times it: the Values cost far more than the floats, and backward costs about as much as the forward pass that built them.

Time 2,500 products and sums as plain floats, as Values, and backward through them.

# volatile: timing
import math
import random
import time
from ember.scalar import Value


def band(ratio):
    """Round a ratio to the nearest of 1, 3, 10, 30, 100, ... on a log scale."""
    marks = [1, 3, 10, 30, 100, 300, 1000]
    return min(marks, key=lambda m: abs(math.log(ratio) - math.log(m)))


random.seed(0)
n = 2500
ws = [random.uniform(-1, 1) for _ in range(n)]
xs = [random.uniform(-1, 1) for _ in range(n)]

start = time.perf_counter()
plain = sum(w * x for w, x in zip(ws, xs))
t_floats = time.perf_counter() - start

start = time.perf_counter()
params = [Value(w) for w in ws]
out = Value(0.0)
for p, x in zip(params, xs):
    out = out + p * x
t_forward = time.perf_counter() - start

start = time.perf_counter()
out.backward()
t_backward = time.perf_counter() - start

print(f"{n} products and sums: about {4 * n:,} Values in the graph")
print(f"forward with Values: about {band(t_forward / t_floats)}× the time of plain floats")
print(f"backward: about {band(t_backward / t_forward)}× the time of the forward with Values")

2500 products and sums: about 10,000 Values in the graph
forward with Values: about 100× the time of plain floats
backward: about 1× the time of the forward with Values

Your numbers will differ from run to run and machine to machine; the cell rounds them to a band.

What: in the training script train_l12.py, fill in the three lines of each step of train(model, xs, ys, lr, steps): clear the old gradients, run backward(), and move every parameter by −lr times its gradient. The loss, the accuracy and the yield are written for you.

Why: without a loop that clears, fills and follows the gradients, the network never learns. Train mode draws what train yields on a live chart while your network learns two-moons. From lesson 28, every training loop is this loop with an optimizer in place of the hand-written update.

Train runs until the boundary separates the moons: under 30 seconds on a desktop, and about four times longer on a phone. Stop keeps what it has learned so far.

Your ember trains a neural network.

Each step, which three things must happen between computing the loss and the next step? Think of the cell that never reset its gradients.
Clear the gradients left from the last step with model.zero_grad(), fill them in with loss.backward(), then move every parameter a little against its gradient.
Nearly the code: model.zero_grad(); loss.backward(); for p in model.parameters(): p.data -= lr * p.grad

train(...) is a generator that yields {"step", "loss", "acc"} with steps 0, 1, 2.
For a neuron that predicts x0, XOR's loss is 2.0 and its accuracy 0.5.
After one step, every parameter has moved by -lr times its gradient.
With lr = 0 nothing moves, so step 2's gradients equal step 1's: they are not added up.
20 steps at lr = 0.2 bring an MLP(2, [8, 1]) on XOR below 5% of its starting loss.

"""Lesson 12 training script. Train mode runs train(model, xs, ys, lr, steps) and charts each dict it yields.""" def train(model, xs, ys, lr, steps): """Full-batch gradient descent on the mean squared error, one yield per step. Each yield is {"step": s, "loss": the loss before this step's update, "acc": the fraction of xs whose prediction has the sign of its label}. Labels are +1.0 or -1.0.""" for step in range(steps): preds = [model(x) for x in xs] loss = sum((p - y) ** 2 for p, y in zip(preds, ys)) * (1.0 / len(ys)) # Your code: clear the old gradients, run backward, then move every parameter downhill by lr. acc = sum(1 for p, y in zip(preds, ys) if (p.data > 0) == (y > 0)) / len(ys) yield {"step": step, "loss": loss.data, "acc": acc}Running code needs JavaScript. Above is train_l12.py as this lab starts.

Load the weights your training run saved and draw the boundary it learned.

import random
import hx
from ember.scalar_nn import MLP
from ember_testing import boundary_payload, two_moons

random.seed(0)
xs, ys = two_moons(40, noise=0.1, seed=0)   # the points lab 3 trained on
model = MLP(2, [8, 8, 1])                   # lab 3's model
hx.load_run("l12-lab3", into=model)         # your trained weights, or the saved fallback run
print(f"loaded {len(model.parameters())} parameters; the plot shows the sign of the model's output")
hx.plot("boundary", boundary_payload(lambda point: model(point).data, xs, ys))

loaded 105 parameters; the plot shows the sign of the model's output

Your trained network's output over the plane, with the two-moons points: a curved boundary between the moons.

class	points
0	20
1	20
16×16 grid, 46% of it on the class 1 side

The same three lines each step, on XOR: an MLP(2, [4, 1]) with 4 hidden tanh neurons and one plain output, built from your classes.

import random
from ember.scalar_nn import MLP

XOR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]

random.seed(0)
model = MLP(2, [4, 1])        # 4 hidden tanh neurons, then one plain output: 17 parameters
for step in range(301):
    loss = 0.0
    for x, target in XOR:
        loss = loss + (model(x) - target) ** 2
    loss = loss * 0.25
    if step % 100 == 0:
        print(f"step {step:3}: loss {loss.data:.4f}")
    model.zero_grad()
    loss.backward()
    for p in model.parameters():
        p.data -= 0.1 * p.grad
for x, target in XOR:
    print(f"{x} -> {model(x).data:.2f}   (want {target})")

step   0: loss 0.6766
step 100: loss 0.1840
step 200: loss 0.0582
step 300: loss 0.0048
(0, 0) -> 0.04   (want 0)
(0, 1) -> 0.94   (want 1)
(1, 0) -> 0.93   (want 1)
(1, 1) -> 0.09   (want 0)

After 300 steps the loss is 0.0048, and after the last step the four outputs are 0.04, 0.94, 0.93 and 0.09: XOR, the rule a single neuron could not draw.

How many parameters does an MLP with 3 inputs, two hidden layers of 8 neurons and one output neuron have?

8 · (3 + 1) = 32, then 8 · (8 + 1) = 72, then 1 · (8 + 1) = 9: 113.

You remove every tanh from your MLP, so each layer is a weighted sum plus a bias. What can it still learn?

The same boundaries as before, only more slowly

Without tanh between them, the layers collapse into one, however many there are.

Only straight-line boundaries: stacked linear layers collapse into one

Without tanh between them, the layers compose into a single straight-line function.

Nothing: it can no longer be trained

It trains, to the best straight line.

One straight boundary, as in lesson 05. The tanh between the layers is where the bends come from.

Where this goes

Your engine learns, but every number is a Python object. Part III packs thousands of numbers into one array, and your framework grows from Values into tensors.

When you finish this lesson

Your ember can now: train a multi-layer network on XOR and on two-moons

Part II gave your ember all of this:

record how every number was made, and draw it as a graph
send a gradient backward through + and *
get the right gradient for a value used in many places
backpropagate through any graph with one call to backward()
differentiate pow, exp, log, tanh and relu, and prove it numerically
train a multi-layer network on XOR and on two-moons

Back up your ember (one small zip).

Next Checkpoint A · ~40 min

Continue

Previous 11 Trust, but verify Next Checkpoint A Checkpoint A

Report a problem with this lesson