Lesson 09 · Part II · Autograd on single numbers

Many paths, one sum

A value used twice: why does = instead of += break training silently?

~35 min · 2 labs · builds on 08 Who owes the gradient?

This lesson builds on lessons 01–08. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

$f = ((a + b) \cdot c) \cdot d$ with a = 4, b = −1, c = 2 and d = −6. What is $\frac{\partial f}{\partial a}$ ?

The last product sends $(a + b) \cdot c$ the gradient d = −6. That product sends $a + b$ the gradient −6 · c = −12, and the sum passes −12 to a unchanged. Local slopes multiply along the path: 1 · 2 · (−6) = −12.

Measuring one whole gradient by nudging, with every slope sharing one measurement where you stand, cost 4,097 forward passes. How many knobs does the model have?

4,096: one pass is where you stand, and each of the others nudges one knob. Backpropagation's promise is one pass forward and one back.

A Value used twice

Your + and * now pass the gradient backward: each input receives out.grad times its local slope, which is enough for a graph where every Value is used once.

Real models reuse Values all the time. One weight multiplies the input of every example, and a squared error multiplies a miss by itself. What happens to a Value used twice?

$b = a + a$ , with a = 3. Nudge a by 0.01. By how much does b change per unit of nudge? That is, what is $\frac{\partial b}{\partial a}$ ?

$a + a = 2a$ , so nudging a by 0.01 moves b by 0.02: the slope is 2.

Your lesson 08 backward runs on $b = a + a$ at a = 3: set b.grad = 1 and call b._backward(). What does a.grad hold afterwards?

2.0: the engine sees a used twice and adds both shares

Nothing in the engine adds: both lines of +'s backward write the same a.grad, and the second replaces the first.

1.0: the second write replaces the first

+'s backward sets self.grad and then other.grad, and here both are a: the second line overwrites the first.

0.5: a used twice gets half from each use

Each use passes the full gradient of 1; nothing is split, and nothing is added either.

1.0, while the true slope is 2. Both lines of +'s backward target a.grad, and the second overwrites the first. The cell below runs your lesson 08 code on a + a and a * a: it gives 1.0 for a + a, where nudging gives 2.00, and 3.0 for a * a, where nudging gives 6.01.

Your lesson 08 code on a + a and a * a, next to the nudged slopes.

# ember: 08
from ember.scalar import Value

a = Value(3.0)
b = a + a
b.grad = 1.0
b._backward()
nudged = ((3.01 + 3.01) - (3.0 + 3.0)) / 0.01
print(f"b = a + a at a = 3: backward gives {a.grad}, nudging gives {nudged:.2f}")

a = Value(3.0)
c = a * a
c.grad = 1.0
c._backward()
nudged = (3.01 * 3.01 - 3.0 * 3.0) / 0.01
print(f"c = a * a at a = 3: backward gives {a.grad}, nudging gives {nudged:.2f}")

b = a + a at a = 3: backward gives 1.0, nudging gives 2.00
c = a * a at a = 3: backward gives 3.0, nudging gives 6.01

No error and no warning: a model trained on these gradients would learn worse, and nothing would tell you.

Every path pays in

When a Value feeds several places, a nudge to it travels down every path at once, and each path moves the output by its own share. So the Value's gradient is the sum of what every path sends back.

Take $c = a \cdot b + a$ with a = 2 and b = 5. Through $a \cdot b$ , a receives b = 5; through the + directly, it receives 1; its gradient is 5 + 1 = 6. Nudging agrees: a = 2.01 gives c = 2.01 · 5 + 2.01 = 12.06 instead of 12, a slope of 6.

So each _backward must add its share into an input's grad with +=, instead of setting it with =. This adding up is called gradient accumulation. Adding needs a starting point, and it has one: a new Value's grad starts at 0.

The graph of $c = a \cdot b + a$ at a = 2 and b = 5, walked backward from c. The Value a feeds both the product and the sum, so two shares arrive at a: 1 straight from the sum and 5 through the product, 6 in all.

node	op	forward value	gradient
a	input	2	6
b	input	5	2
a*b	*	10	1
c	+	12	1

$d = a \cdot b + b$ with a = 4 and b = −2. What is $\frac{\partial d}{\partial b}$ ?

Through $a \cdot b$ , b receives a = 4; through the + directly, 1. The sum is 5.

Subtraction needs no backward of its own: $a - b = a + b \cdot (-1)$ , so b receives −1 through a product and a sum you already have.

What: change the _backward of + and * so each adds its share into each input's grad (+=) instead of setting it.

Why: every weight in a real model is used many times. Without this, lesson 12's network trains on wrong gradients and no error ever says so.

Your ember sums the gradient of a Value used in many places.

A Value used twice receives two shares. Which assignment operator keeps the first share when the second arrives?
In both _backward functions, turn every = that writes a .grad into +=. Nothing else changes.
Nearly the code: self.grad += other.data * out.grad and other.grad += self.data * out.grad.

In a + a both uses send 1, so a's gradient is 2.
In a * a each use sends the other's value, so a's gradient is 2a.
In y = a*b + a, a receives b from the product and 1 from the sum.
A gradient already in a.grad is kept and added to.

"""Part II: Value, a number that remembers how it was made (the scalar autograd engine).""" from collections.abc import Callable # A type hint that names Value is written in quotes, "Value": this def runs before class Value # exists, and inside the class the name is not ready until the class body ends. Python keeps a # quoted hint as text instead of looking the name up. Value | float means a Value or a plain # number, and tuple["Value", ...] is a tuple of Values of any length. def _as_value(x: "Value | float") -> "Value": """A Value for x: x itself when it already is one, else a new leaf Value holding the number. Args: x: A Value, or a plain number such as 2 or 0.5. Returns: x itself when it is a Value, else a new leaf Value whose data is x as a float. """ if isinstance(x, Value): return x return Value(x) class Value: """A number that remembers the Values it was made from (_prev) and the operation (_op). Attributes: data: The number, as a float. grad: How much the final result (such as a loss) changes for each unit that data grows; 0.0 until a backward pass fills it in. _prev: The Values this one was made from, its parents, as a tuple; () for a leaf. _op: The symbol of the operation that made this Value, such as "+"; "" for a leaf. _backward: Passes this Value's grad back to its parents. It does nothing for a leaf, or for a Value whose operation has no backward yet. """ def __init__(self, data: float, _children: tuple["Value", ...] = (), _op: str = "") -> None: """Make a Value holding data. Args: data: The number, such as 2 or 0.5; it is kept as a float. _children: The Values this one was made from, its parents; () for a leaf. _op: The symbol of the operation that made it, such as "+"; "" for a leaf. """ # --- gap L07.init --- self.data = float(data) self.grad = 0.0 self._prev = tuple(_children) self._op = _op # --- end --- self._backward: Callable[[], None] = lambda: None def __repr__(self) -> str: """How print shows a Value: its data and its grad. Returns: Text such as Value(2.0, grad=0.0). """ # --- gap L07.repr --- return f"Value({self.data!r}, grad={self.grad!r})" # --- end --- def __add__(self, other: "Value | float") -> "Value": """self + other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the sum, with self and other (as a Value) as its parents and "+" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a + a, ends up with both shares. """ # --- gap L07.add --- other = _as_value(other) out = Value(self.data + other.data, (self, other), "+") # --- end --- # --- gap L09.add_backward --- def _backward(): self.grad = out.grad other.grad = out.grad out._backward = _backward # --- end --- return out def __mul__(self, other: "Value | float") -> "Value": """self * other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the product, with self and other (as a Value) as its parents and "*" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a * a, ends up with both shares. """ # --- gap L07.mul --- other = _as_value(other) out = Value(self.data * other.data, (self, other), "*") # --- end --- # --- gap L09.mul_backward --- def _backward(): self.grad = other.data * out.grad other.grad = self.data * out.grad out._backward = _backward # --- end --- return out # Lesson 07 gives these five forward only; lesson 09 rebuilds them from + and *. def __neg__(self) -> "Value": """-self, as a new Value. Returns: A new Value holding minus this Value's data. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ # --- gap L09.neg --- return Value(-self.data, (self,), "neg") # --- end --- def __sub__(self, other: "Value | float") -> "Value": """self - other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches both sides through it: per unit of the result's grad, self gets 1 and other, when it is a Value, gets -1. """ # --- gap L09.sub --- other = _as_value(other) return Value(self.data - other.data, (self, other), "-") # --- end --- def __rsub__(self, other: "Value | float") -> "Value": """other - self, when the left side is a plain number, as in 2 - a. Args: other: The left side of the minus, a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ # --- gap L09.rsub --- other = _as_value(other) return Value(other.data - self.data, (other, self), "-") # --- end --- def __radd__(self, other: "Value | float") -> "Value": """other + self, when the left side is a plain number, as in 2 + a. Args: other: The left side of the plus, a plain number such as 2. Returns: A new Value holding the sum. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets 1 per unit of the result's grad. """ # --- gap L09.radd --- other = _as_value(other) return Value(other.data + self.data, (other, self), "+") # --- end --- def __rmul__(self, other: "Value | float") -> "Value": """other * self, when the left side is a plain number, as in 2 * a. Args: other: The left side of the times, a plain number such as 2. Returns: A new Value holding the product. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets other per unit of the result's grad. """ # --- gap L09.rmul --- other = _as_value(other) return Value(other.data * self.data, (other, self), "*") # --- end --- # Lesson 07 gives division forward only; lesson 11 rebuilds it from * and **. def __truediv__(self, other: "Value | float") -> "Value": """self / other, as a new Value. Args: other: A Value, or a plain number such as 2; not 0. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If other is 0. """ other = _as_value(other) return Value(self.data / other.data, (self, other), "/") def __rtruediv__(self, other: "Value | float") -> "Value": """other / self, when the left side is a plain number, as in 2 / a. Args: other: The left side of the division, a plain number such as 2. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If this Value's data is 0. """ other = _as_value(other) return Value(other.data / self.data, (other, self), "/") def __pow__(self, exponent: float) -> "Value": """self ** exponent, as a new Value. The exponent is a plain number, never a Value. Args: exponent: An int or a float, such as 2 or -1. Returns: A new Value holding this Value's data to the power exponent, with this Value as its parent and "**" followed by the exponent, such as "**2", as its operation. Raises: TypeError: If exponent is not an int or a float (for example a Value), or if data is negative and exponent is not a whole number. ZeroDivisionError: If this Value's data is 0 and exponent is negative. OverflowError: If the result is too large for a float. """ if not isinstance(exponent, (int, float)): raise TypeError("Value ** needs an int or float exponent, not a Value") # Forward only until lesson 11 gives ** its backward. return Value(self.data ** exponent, (self,), f"**{exponent}")Running code needs JavaScript. Above is ember/scalar.py as this lab starts.

What: build -x, a - b and the reflected forms, such as 2 + a, 2 * a and 2 - a, out of + and *, so none of them needs a backward of its own.

Why: without it, a loss that subtracts a prediction from a target has no backward at all, because lesson 07 gave - a forward only. Built from + and *, subtraction is handled by lesson 10's backward with no new code.

Your ember can subtract, and backpropagate through it.

Can you write -x using only *? And a - b using only + and a negation?
-x is x * -1. a - b is a + (-b). 2 + a is a + 2, 2 * a is a * 2, and 2 - a is (-a) + 2.
Nearly the code: __neg__ returns self * -1, __sub__ returns self + (-other), and __rsub__ returns other + (-self).

-a has data -a and sends -1 back to a.
In a - b, a receives 1 and b receives -1.
2 + a, 3 * a and 5 - a compute and pass gradients to a.
a - 2 computes and passes 1 back to a.

The two graphs from before with your new code, and a - b.

from ember.scalar import Value

a = Value(3.0)
b = a + a
b.grad = 1.0
b._backward()
print(f"a + a at a = 3: a.grad = {a.grad}")

a = Value(3.0)
c = a * a
c.grad = 1.0
c._backward()
print(f"a * a at a = 3: a.grad = {a.grad}")

a, b = Value(5.0), Value(2.0)
d = a - b
d.grad = 1.0
d._backward()                 # the + inside a - b
for p in d._prev:
    p._backward()             # the * -1 inside -b (a leaf's _backward does nothing)
print(f"a - b: a.grad = {a.grad}, b.grad = {b.grad}")

a + a at a = 3: a.grad = 2.0
a * a at a = 3: a.grad = 6.0
a - b: a.grad = 1.0, b.grad = -1.0

The cell prints 2.0 for a + a, 6.0 for a * a, and 1.0 and −1.0 for a − b: the true slopes, now from backward.

$f = a \cdot a \cdot a$ at a = 2, computed as $(a \cdot a) \cdot a$ . Once both products have passed their shares back, what gradient does a hold?

The outer product sends a the value of $a \cdot a$ , 4. It sends $a \cdot a$ the value a = 2, and $a \cdot a$ passes 2 · 2 = 4 to each of its two uses of a. Together 4 + 4 + 4 = 12.

A weight w multiplies the input of each of 64 examples in one loss. What gradient should w end up with?

The share from one of the uses: a Value gets its gradient from one place

Every use changes the loss when w changes, so every use contributes to the slope.

The average of the 64 shares, which the engine works out

The engine only adds. If the loss averages over the examples, as mse does, that division reaches every share through the chain rule.

The sum of the 64 shares, one from each use

w is one Value used 64 times: every use sends back a share, and backward adds them.

The sum. Averaging, when you want it, belongs in the loss, not in the engine.

Where this goes

With shared Values, order matters too: a Value must collect every share before it passes gradient on to its own parents. Lesson 10 finds that order.

When you finish this lesson

Your ember can now: get the right gradient for a value used in many places

It already could:

send a gradient backward through + and *
record how every number was made, and draw it as a graph
measure every slope of a loss at once, and count the cost

Next The order of blame · In what order must the graph be visited? · ~40 min

Continue

Previous 08 Who owes the gradient?Next 10 The order of blame In what order must the graph be visited?

Report a problem with this lesson