Lesson 08 · Part II · Autograd on single numbers

Who owes the gradient?

Each op knows only its local slope. Is that enough?

~40 min · 1 lab · builds on 07 A number that remembers

This lesson builds on lessons 01–07. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

You write a = Value(2.0), b = Value(3.0) and c = a + b, then set a.data = 10.0. What data do the two parents in c._prev hold now?

2.0 and 3.0: _prev kept copies of the numbers when c was made

_prev keeps the parent Values, not their numbers: backward needs the objects to write gradients into.

10.0 and 3.0: _prev holds a itself, so it sees the change

c records a and b as objects, so reading a through c's record reads the a you changed.

10.0 and 3.0. _prev holds the parent Values themselves, the objects a backward pass writes gradients into. c.data is still 5.0: nothing recomputes it until you add again. Today you walk those links backward.

Five straight-line layers in a row, with no kink between them. How many straight-line layers could replace them exactly?

One. A line of a line is a line, however many you stack: the slopes multiply and the offsets carry through.

Three suspects

A Value records the operation that made it and the Values it came from, so a loss built from Values carries its whole history.

Here is one. w = 0.5, x = 2 and b = 1 make the prediction $z = w \cdot x + b$ = 2. The target is y = 3, so the loss $(z - y)^{2}$ is 1.

Nudge z itself from 2 to 2.01 and the loss becomes $(2.01 - 3)^{2}$ = 0.9801, down by 0.0199: a measured slope of −1.99, about −2 per unit of z.

Three numbers, w, x and b, made that miss. Which should change, and by how much? Nudging each would cost three more passes. Today every answer comes from one pass backward.

Nudge w from 0.50 to 0.51. Roughly how much does the loss change?

+0.01: the loss moves one-for-one with w

Try it: w = 0.51 gives z = 2.02 and a loss of 0.9604. The loss went down, and by more than 0.01.

−0.02: w's nudge becomes z's nudge, and z's slope is −2

That is what nudging z directly would do. But w reaches z through a multiplication by x = 2, so its effect on z is twice as large.

−0.04: x = 2 doubles w's nudge on its way into z, and z's slope is −2

The nudge is multiplied by x = 2 on its way into z, and each unit of z moves the loss by −2: 0.01 · 2 · (−2) = −0.04.

0.00: the slopes along the way, 2 and −2, cancel

That adds the slopes. Along a path they multiply: 2 · (−2) = −4.

About −0.04. Each step along the path multiplies the nudge by that step's local slope. The cell below measures it: the loss goes from 1.0000 to 0.9604, a change of −0.0396, almost exactly 0.01 · 2 · (−2) = −0.04.

Nudge w by 0.01 and measure: one extra forward pass for one knob.

def loss(w, x=2.0, b=1.0, y=3.0):
    z = w * x + b
    return (z - y) ** 2


before, after = loss(0.50), loss(0.51)
print(f"loss at w = 0.50: {before:.4f}")
print(f"loss at w = 0.51: {after:.4f}")
print(f"change: {after - before:+.4f}")

loss at w = 0.50: 1.0000
loss at w = 0.51: 0.9604
change: -0.0396

Rates multiply

Each operation knows only its local slope: how fast its output moves per unit of one input, with the other input held still. For a sum $a + b$ , each input's local slope is 1. For a product $a \cdot b$ , the local slope for a is b, and for b it is a. The square at the top has no rule yet; lesson 11 derives one. Its local slope here is the −2 you measured by nudging z.

Going backward, each operation multiplies the gradient arriving from above by its local slope, and hands the result to its inputs. That is the chain rule; applied backward through a whole graph, it is called backpropagation.

Along the path from w to L the local slopes multiply: $\frac{\partial L}{\partial w} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial w}$ which here is (−2) · 2 = −4, at w = 0.5, x = 2, b = 1 and y = 3.

Same graph. What is $\frac{\partial L}{\partial b}$ ?

(−2) · 1 = −2: a sum passes the gradient through unchanged.

And $\frac{\partial L}{\partial x}$ ?

(−2) · w = (−2) · 0.5 = −1: a product sends each input the gradient times the other input.

The graph of $z = w \cdot x + b$ , walked backward from z's gradient of −2. Step through it: each node shows its value and the gradient it receives.

node	op	forward value	gradient
b	input	1	-2
x	input	2	-1
w	input	0.5	-4
w*x	*	1	-2
z	+	2	-2

What: fill in the _backward of + and *: + sets each input's grad to out.grad; * sets each input's grad to out.grad times the other input's data.

Why: without a backward, every slope still costs one forward pass per knob. Every sum and product in every model you build later runs these lines. Lesson 09 fixes what happens when an input is used twice, and lesson 10 runs them in the right order for you.

Your ember can send a gradient backward through + and *.

In out = self + other, if self grows by a tiny amount, how much does out grow? And in out = self * other?
For +, each input's local slope is 1, so each gets out.grad. For *, self's local slope is other.data and other's is self.data; multiply each by out.grad, which is the chain rule.
Nearly the code: in *'s _backward, self.grad = other.data * out.grad and other.grad = self.data * out.grad; in +'s, self.grad = out.grad and other.grad = out.grad.

+ has local slope 1 for both inputs, so each receives out.grad.
For a * b, a's gradient is b times out.grad and b's is a times out.grad.
Running _backward from L down to the leaves gives every gradient of L = (w*x) * (u + v).
Calling _backward on a leaf is allowed and changes nothing.
In a * 4 the gradient that reaches a is 4.

"""Part II: Value, a number that remembers how it was made (the scalar autograd engine).""" from collections.abc import Callable # A type hint that names Value is written in quotes, "Value": this def runs before class Value # exists, and inside the class the name is not ready until the class body ends. Python keeps a # quoted hint as text instead of looking the name up. Value | float means a Value or a plain # number, and tuple["Value", ...] is a tuple of Values of any length. def _as_value(x: "Value | float") -> "Value": """A Value for x: x itself when it already is one, else a new leaf Value holding the number. Args: x: A Value, or a plain number such as 2 or 0.5. Returns: x itself when it is a Value, else a new leaf Value whose data is x as a float. """ if isinstance(x, Value): return x return Value(x) class Value: """A number that remembers the Values it was made from (_prev) and the operation (_op). Attributes: data: The number, as a float. grad: How much the final result (such as a loss) changes for each unit that data grows; 0.0 until a backward pass fills it in. _prev: The Values this one was made from, its parents, as a tuple; () for a leaf. _op: The symbol of the operation that made this Value, such as "+"; "" for a leaf. _backward: Passes this Value's grad back to its parents. It does nothing for a leaf, or for a Value whose operation has no backward yet. """ def __init__(self, data: float, _children: tuple["Value", ...] = (), _op: str = "") -> None: """Make a Value holding data. Args: data: The number, such as 2 or 0.5; it is kept as a float. _children: The Values this one was made from, its parents; () for a leaf. _op: The symbol of the operation that made it, such as "+"; "" for a leaf. """ # --- gap L07.init --- self.data = float(data) self.grad = 0.0 self._prev = tuple(_children) self._op = _op # --- end --- self._backward: Callable[[], None] = lambda: None def __repr__(self) -> str: """How print shows a Value: its data and its grad. Returns: Text such as Value(2.0, grad=0.0). """ # --- gap L07.repr --- return f"Value({self.data!r}, grad={self.grad!r})" # --- end --- def __add__(self, other: "Value | float") -> "Value": """self + other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the sum, with self and other (as a Value) as its parents and "+" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a + a, ends up with both shares. """ # --- gap L07.add --- other = _as_value(other) out = Value(self.data + other.data, (self, other), "+") # --- end --- # --- gap L08.add_backward --- def _backward(): raise NotImplementedError("Set self.grad and other.grad from out.grad and the local slope of +.") out._backward = _backward # --- end --- return out def __mul__(self, other: "Value | float") -> "Value": """self * other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the product, with self and other (as a Value) as its parents and "*" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a * a, ends up with both shares. """ # --- gap L07.mul --- other = _as_value(other) out = Value(self.data * other.data, (self, other), "*") # --- end --- # --- gap L08.mul_backward --- def _backward(): raise NotImplementedError("Set self.grad and other.grad from out.grad and the local slopes of *.") out._backward = _backward # --- end --- return out # Lesson 07 gives these five forward only; lesson 09 rebuilds them from + and *. def __neg__(self) -> "Value": """-self, as a new Value. Returns: A new Value holding minus this Value's data. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ return Value(-self.data, (self,), "neg") def __sub__(self, other: "Value | float") -> "Value": """self - other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches both sides through it: per unit of the result's grad, self gets 1 and other, when it is a Value, gets -1. """ other = _as_value(other) return Value(self.data - other.data, (self, other), "-") def __rsub__(self, other: "Value | float") -> "Value": """other - self, when the left side is a plain number, as in 2 - a. Args: other: The left side of the minus, a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ other = _as_value(other) return Value(other.data - self.data, (other, self), "-") def __radd__(self, other: "Value | float") -> "Value": """other + self, when the left side is a plain number, as in 2 + a. Args: other: The left side of the plus, a plain number such as 2. Returns: A new Value holding the sum. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets 1 per unit of the result's grad. """ other = _as_value(other) return Value(other.data + self.data, (other, self), "+") def __rmul__(self, other: "Value | float") -> "Value": """other * self, when the left side is a plain number, as in 2 * a. Args: other: The left side of the times, a plain number such as 2. Returns: A new Value holding the product. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets other per unit of the result's grad. """ other = _as_value(other) return Value(other.data * self.data, (other, self), "*") # Lesson 07 gives division forward only; lesson 11 rebuilds it from * and **. def __truediv__(self, other: "Value | float") -> "Value": """self / other, as a new Value. Args: other: A Value, or a plain number such as 2; not 0. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If other is 0. """ other = _as_value(other) return Value(self.data / other.data, (self, other), "/") def __rtruediv__(self, other: "Value | float") -> "Value": """other / self, when the left side is a plain number, as in 2 / a. Args: other: The left side of the division, a plain number such as 2. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If this Value's data is 0. """ other = _as_value(other) return Value(other.data / self.data, (other, self), "/") def __pow__(self, exponent: float) -> "Value": """self ** exponent, as a new Value. The exponent is a plain number, never a Value. Args: exponent: An int or a float, such as 2 or -1. Returns: A new Value holding this Value's data to the power exponent, with this Value as its parent and "**" followed by the exponent, such as "**2", as its operation. Raises: TypeError: If exponent is not an int or a float (for example a Value), or if data is negative and exponent is not a whole number. ZeroDivisionError: If this Value's data is 0 and exponent is negative. OverflowError: If the result is too large for a float. """ if not isinstance(exponent, (int, float)): raise TypeError("Value ** needs an int or float exponent, not a Value") # Forward only until lesson 11 gives ** its backward. return Value(self.data ** exponent, (self,), f"**{exponent}")Running code needs JavaScript. Above is ember/scalar.py as this lab starts.

The graph above, backward by hand with your code: set z's gradient, then call each _backward from the top down. Running it redraws that graph from your Values, with their gradients; step through it to see them.

import hx
from ember.scalar import Value

w, x, b = Value(0.5), Value(2.0), Value(1.0)
for v, name in [(w, "w"), (x, "x"), (b, "b")]:
    v.label = name
wx = w * x
wx.label = "w*x"
z = wx + b
z.label = "z"


def squared_miss(value):
    return (value - 3.0) ** 2     # the loss as a function of z, with the target y = 3


z.grad = round((squared_miss(z.data + 0.000001) - squared_miss(z.data)) / 0.000001, 3)   # measured by nudging z
z._backward()                 # + passes z's gradient to w*x and to b
wx._backward()                # * passes w*x's gradient to w and to x
print(f"z.grad = {z.grad}, (w*x).grad = {wx.grad}")
print(f"w.grad = {w.grad}, x.grad = {x.grad}, b.grad = {b.grad}")
hx.show_graph(z)

z.grad = -2.0, (w*x).grad = -2.0
w.grad = -4.0, x.grad = -1.0, b.grad = -2.0

From one pass backward, the cell prints w.grad = −4.0, x.grad = −1.0 and b.grad = −2.0: the three answers to the questions above.

$f = (a + b) \cdot c$ with a = 1, b = 2 and c = −3. What is $\frac{\partial f}{\partial a}$ ?

The product sends a + b the gradient c = −3, and the sum passes it through unchanged: −3. For c it is a + b = 3.

$g = (a \cdot b) \cdot c$ with a = 2, b = 3 and c = 4. What is $\frac{\partial g}{\partial a}$ ?

7: b plus c

That adds the local slopes. Along a path they multiply.

12: c times b

The gradient reaching a · b is c = 4, and a · b passes it to a times b = 3: 4 · 3 = 12.

4: the slope of the last multiplication, c

4 is the slope of g with respect to a · b; a reaches g through one more multiplication, by b = 3.

12. Check it by nudging: a = 2.01 gives 2.01 · 3 · 4 = 24.12, up 0.12 for a nudge of 0.01.

Where this goes

Your backward assumes each Value is used once. Use one twice, as in a * a, and it gives the wrong gradient without any error. Lesson 09 finds out why.

When you finish this lesson

Your ember can now: send a gradient backward through + and *

It already could:

record how every number was made, and draw it as a graph
measure every slope of a loss at once, and count the cost
compute XOR with two layers and a kink

Next Many paths, one sum · A value used twice: why does = instead of += break training silently? · ~35 min

Continue

Previous 07 A number that remembers Next 09 Many paths, one sum A value used twice: why does = instead of += break training silently?

Report a problem with this lesson