Lesson 11 · Part II · Autograd on single numbers

Trust, but verify

How do you know your derivative is right?

~45 min · 2 labs · builds on 10 The order of blame

This lesson builds on lessons 01–10. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

m = a * b and q = m + a. You call each node's _backward by hand. Which order gives every Value its full gradient?

m first, then q: m is closer to the inputs, so it goes first

m would pass its gradient on before q had sent m anything, so a would get nothing along that path.

q first, then m: q uses m, so q must hand m its share before m passes anything on

Users before the used: m's gradient is complete only after q has run.

q, then m. Backward runs from the output toward the inputs, users before the used.

You could check a backward by nudging, the way lesson 6 measured slopes. Why do checks nudge small functions with a few inputs, not a whole network?

They would work as well on a whole network; small functions are only easier to read

At 1 ms a pass, a million passes take about 17 minutes, and that checks one gradient.

Nudging costs a forward pass per input, so checking a big network would cost as much as training it the slow way

Each knob needs its own nudged pass, so one check of a million-knob network costs at least a million passes.

Cost. Check each operation once, on a few inputs; backpropagation then composes operations you trust.

A wrong slope doesn't crash

Your Value.backward() walks any graph of + and * in topological order and delivers every gradient in one pass.

A network also needs powers, exp, log, tanh and relu, each with its own local slope. A wrong formula raises nothing: the model trains badly, and nobody knows why. How do you know a derivative is right?

Your backward says 2000.000 and nudging says 2000.001. Elsewhere, backward says 0.002 and nudging says 0.001. Both disagree by 0.001. Which disagreement points to a bug?

Both, equally: the gap is the same

The same gap means very different things next to 2000 and next to 0.002.

The second: it is off by half of its size

0.001 out of 0.002 is a factor of two; 0.001 out of 2000 is what nudging and rounding leave behind.

The first: bigger numbers deserve more suspicion

Big gradients carry big rounding; what matters is the gap compared with the size.

The second. Judge a gap by its size relative to the numbers, a rule the derivation below makes exact.

Before any formula, measure. numerical_derivative, which nudges one input and divides, still works on any function of one number. One of the five functions below, tanh, is a smooth step from −1 to 1, like sigmoid's from 0 to 1.

Measure five local slopes by nudging: $x^{3}$ , exp, the natural log ln, tanh and relu.

import math
from ember.basics import numerical_derivative


def relu(x):
    return max(0.0, x)


for name, f, x in [("x**3", lambda t: t ** 3, 2.0), ("exp", math.exp, 0.5), ("ln", math.log, 2.0),
                   ("tanh", math.tanh, 0.0), ("relu", relu, -1.0), ("relu", relu, 2.0)]:
    print(f"slope of {name:4} at x = {x:4}: {numerical_derivative(f, x):.4f}")

slope of x**3 at x =  2.0: 12.0000
slope of exp  at x =  0.5: 1.6487
slope of ln   at x =  2.0: 0.5000
slope of tanh at x =  0.0: 1.0000
slope of relu at x = -1.0: 0.0000
slope of relu at x =  2.0: 1.0000

Measured: 12.0000 for $x^{3}$ at 2, 1.6487 for exp at 0.5, 0.5000 for ln at 2, 1.0000 for tanh at 0, and 0.0000 and 1.0000 for relu at −1 and at 2.

Here is a cheat sheet of formulas for those slopes, of the kind you might find anywhere:

function	local slope
$x^{n}$	$n x^{n-1}$
$e^{x}$	$e^{x}$
$\ln x$	$\frac{1}{x}$
$\tanh x$	$1 - \tanh x$
$\mathrm{relu}(x)$	1 when x > 0, 0 when x < 0

At x = 0 the cheat sheet's tanh line gives 1 − tanh(0) = 1, and the measurement gives 1.0000. What does that tell you about the formula?

It is right: formula and measurement agree

They agree at one point. Many different formulas pass through the same point.

Only that it gives the right value at x = 0: one point can't confirm a formula

A formula is a claim about every x; one measurement tests it at one x.

Nothing: nudging can't check a formula

It can, at every point you try, which is why a check tries more than one.

One point only. A check has to try several points, and it needs a fair way to judge a disagreement. Keep an eye on that tanh line.

The table, measured

Go down the table one row at a time, each next to its measurement. One function is new: the natural log, written ln, which undoes $e^{x}$ . ember names its method log. Leave the tanh row for the checker you are about to build.

Powers: the slope of $x^{n}$ is $n x^{n-1}$ , and at x = 2 with n = 3 that is $3 \cdot 2^{2}$ = 12, as measured. The exponential is its own slope: $e^{0.5} \approx 1.6487$ , as measured.

The natural log undoes the exponential, $\ln e^{x} = x$ , and its slope is $\frac{1}{x}$ : at x = 2 that is 0.5, as measured. relu's slope is 0 on the left of 0 and 1 on the right, as measured.

The slope of $\ln x$ is $\frac{1}{x}$ . What is the slope of ln at x = 4?

1 / 4 = 0.25: the log flattens as x grows.

The measurement itself can be better. A one-sided nudge measures the slope a little to the right of x. Nudge both ways instead and divide by 2h, and the errors on the two sides cancel. That is the central difference.

The central difference nudges both ways: $\frac{\partial f}{\partial x} \approx \frac{f(x + h) - f(x - h)}{2h}$ For $x^{3}$ at x = 1 with h = 0.01 that is (1.030301 − 0.970299) / 0.02 = 3.0001, off by 0.0001, where the one-sided nudge gives 3.0301, off by 0.0301.

How far off each estimate of the slope of $x^{3}$ at 1 is, for nudges from 0.01 down to 0.000000000001.

print("The slope of x**3 at x = 1 is exactly 3. How far off is each estimate?")
for h in [1e-2, 1e-4, 1e-6, 1e-8, 1e-10, 1e-12]:
    forward = ((1 + h) ** 3 - 1) / h
    central = ((1 + h) ** 3 - (1 - h) ** 3) / (2 * h)
    print(f"h = {h:.0e}: forward off by {abs(forward - 3):.1e}, central off by {abs(central - 3):.1e}")

The slope of x**3 at x = 1 is exactly 3. How far off is each estimate?
h = 1e-02: forward off by 3.0e-02, central off by 1.0e-04
h = 1e-04: forward off by 3.0e-04, central off by 1.0e-08
h = 1e-06: forward off by 3.0e-06, central off by 8.0e-11
h = 1e-08: forward off by 4.0e-09, central off by 7.1e-09
h = 1e-10: forward off by 2.5e-07, central off by 2.5e-07
h = 1e-12: forward off by 2.7e-04, central off by 1.0e-04

Smaller nudges help until rounding takes over. The central difference is best near h = 1e-06 (Python's way of writing 0.000001), off by 8.0e-11. At h = 1e-12 rounding has undone the gain: it is off by 1.0e-04 again, as at h = 0.01.

To judge a disagreement, divide the gap by the size: $\frac{\lvert a - b \rvert}{\max(\lvert a \rvert, \lvert b \rvert)}$ For 2000.000 against 2000.001 that is 0.001 / 2000.001 ≈ 0.0000005; for 0.002 against 0.001 it is 0.001 / 0.002 = 0.5.

Backward says 0.30 and nudging says 0.25. By the formula above, what is the relative error?

|0.30 − 0.25| / 0.30 = 0.05 / 0.30 ≈ 0.1667: a bug, whatever the absolute gap looks like.

What: write numerical_grad(func, inputs, eps), the central-difference slope of func, a function of a list of plain numbers, for each input. Then write gradcheck_scalar(func, inputs, eps, atol, rtol): run func on a list of Values, call backward(), and return True only if every input's grad is within atol + rtol times the measured size of numerical_grad's slope. That is the relative error above, multiplied out so nothing divides by zero, with the measured size in place of the larger one (the two are nearly equal when the check passes) and a small absolute floor.

Why: without it, a wrong local slope trains a model badly and nothing says so. It proves each new backward right before anything trusts it: the cell after it uses it to catch the cheat sheet's tanh, lesson 14 turns it into gradcheck for tensors, and every operation after that is checked by it.

Your ember can check its own derivatives.

numerical_grad nudges one input at a time, both up and down. For gradcheck_scalar: which two numbers do you compare for each input, and how do you make the comparison fair for big and small gradients?
numerical_grad: for each input, copy the list twice, add eps to one copy and subtract it from the other, and divide the difference of func by 2 * eps. gradcheck_scalar: make Values from inputs, call backward on func(values), measure with numerical_grad on a float version of func, and accept each pair when the gap is at most atol + rtol * abs(numeric).
Nearly the code: numeric = numerical_grad(lambda xs: func([Value(x) for x in xs]).data, inputs, eps); return all(abs(v.grad - n) <= atol + rtol * abs(n) for v, n in zip(values, numeric)).

The gradient of x0^2 * x1 at [3, 2] is [12, 9].
With eps = 0.001, x^3 at 1 gives (1.001^3 - 0.999^3) / 0.002 = 3.000001.
numerical_grad does not change the list it is given.
x0 * x1 + e^x0 has a correct backward, so gradcheck_scalar returns True.
An op whose backward sends v instead of 2v fails the check.
A wrong gradient on the second input is caught too.
x^3 at 200 has gradient 120,000: a correct backward passes although the estimate is off by about 0.0004.

"""Checking gradients by nudging: numerical_grad and gradcheck_scalar (lesson 11).""" from collections.abc import Callable from .scalar import Value def numerical_grad( func: Callable[[list[float]], float], inputs: list[float], eps: float = 1e-6, ) -> list[float]: """Every partial slope of func (a function of a list of floats) at inputs, by central differences, one input at a time: (func(x + eps) - func(x - eps)) / (2 * eps). Args: func: Takes a list of floats, as many as inputs holds, and returns a float. inputs: Where to measure. The caller's list is not changed. eps: The size of the nudge: small, and not 0. Returns: A new list with one slope per input, in the order of inputs. Raises: ZeroDivisionError: If eps is 0. """ # --- gap L11.numerical_grad --- raise NotImplementedError("Nudge each input up and down by eps and divide the difference by 2 * eps.") # --- end --- def gradcheck_scalar( func: Callable[[list[Value]], Value], inputs: list[float], eps: float = 1e-6, atol: float = 1e-5, rtol: float = 1e-3, ) -> bool: """True when backward's gradient of func agrees with numerical_grad at inputs for every input. func takes a list of Values and returns a Value. Input i agrees when |analytic_i - numeric_i| <= atol + rtol * |numeric_i|, a tolerance that grows with the size of the gradient. Returns False otherwise. Args: func: Takes a list of Values, as many as inputs holds, and returns one Value built from them with Value operations. inputs: The plain numbers to check at. eps: The nudge handed to numerical_grad. atol: The fixed part of the tolerance; 0 or more. rtol: The part of the tolerance that grows with |numeric_i|; 0 or more. Returns: True when every input agrees, else False. """ # --- gap L11.gradcheck_scalar --- raise NotImplementedError("Compare backward's gradient with numerical_grad for every input, within atol + rtol * |numeric|.") # --- end ---Running code needs JavaScript. Above is ember/gradcheck.py as this lab starts.

Your checker on the cheat sheet's tanh, at x = 0, 0.5 and 1.

import math
from ember.scalar import Value
from ember.gradcheck import gradcheck_scalar


def cheat_sheet_tanh(values):
    """tanh of the one input, with the cheat sheet's backward: local slope 1 - tanh(x)."""
    v = values[0]
    t = math.tanh(v.data)
    out = Value(t, (v,), "tanh")

    def _backward():
        v.grad += (1 - t) * out.grad
    out._backward = _backward
    return out


for x in [0.0, 0.5, 1.0]:
    print(f"x = {x}: gradcheck_scalar says {gradcheck_scalar(cheat_sheet_tanh, [x])}")

x = 0.0: gradcheck_scalar says True
x = 0.5: gradcheck_scalar says False
x = 1.0: gradcheck_scalar says False

The cheat sheet's tanh passes at x = 0 and fails at 0.5 and at 1.0. At x = 1, tanh(1) ≈ 0.7616, so the sheet says 1 − 0.7616 = 0.2384 while the true slope is $1 - 0.7616^{2}$ ≈ 0.42. The right formula is $1 - \tanh^{2} x$ , and the sheet's line was planted for your checker to find.

What: give ** (with a plain-number exponent) its backward by the power rule, and write Value.tanh with its backward, whose local slope is $1 - \tanh^{2} x$ . exp, log and relu arrive ready-made, backward included: read them before you start.

Why: without them, a squared error or a tanh neuron has no backward, and lesson 12's neurons squash with tanh. With ** in place, division is a times b to the power −1 and needs no backward of its own.

Your ember differentiates everything a small network needs, and has checked it.

For out = x ** k, how much does out change per unit change of x? For tanh, remember that the cheat sheet was wrong: check your formula with gradcheck_scalar before you trust it.
Power rule: the local slope of x ** k is k * x ** (k - 1). tanh's local slope is 1 - tanh(x) ** 2; compute t = math.tanh(self.data) once and reuse it. Both add their share into self.grad, times out.grad.
Nearly the code: in __pow__, a _backward that does self.grad += exponent * self.data ** (exponent - 1) * out.grad; in tanh, self.grad += (1 - t * t) * out.grad.

The gradient of a**3 at 2 is 3 * 2**2 = 12.
a**-1 at 4 has gradient -1/16; b**0.5 at 9 has gradient 1/6.
a**2 at -3 has gradient -6.
At 0.5, tanh gives tanh(0.5) and its gradient is 1 - tanh(0.5)**2.
At 0 the gradient of tanh is 1 (a hand check here cannot tell 1 - t from 1 - t^2).
a / b is a * b**-1, so it has gradients 1/b and -a/b^2; 6 / b has -6/b^2.
(a*b + tanh(c))**2 has the gradients the chain rule predicts.
a ** Value(2) raises TypeError: exponents are plain numbers.

"""Part II: Value, a number that remembers how it was made (the scalar autograd engine).""" import math from collections.abc import Callable # A type hint that names Value is written in quotes, "Value": this def runs before class Value # exists, and inside the class the name is not ready until the class body ends. Python keeps a # quoted hint as text instead of looking the name up. Value | float means a Value or a plain # number, and tuple["Value", ...] is a tuple of Values of any length. def _as_value(x: "Value | float") -> "Value": """A Value for x: x itself when it already is one, else a new leaf Value holding the number. Args: x: A Value, or a plain number such as 2 or 0.5. Returns: x itself when it is a Value, else a new leaf Value whose data is x as a float. """ if isinstance(x, Value): return x return Value(x) class Value: """A number that remembers the Values it was made from (_prev) and the operation (_op). Attributes: data: The number, as a float. grad: How much the final result (such as a loss) changes for each unit that data grows; 0.0 until a backward pass fills it in. _prev: The Values this one was made from, its parents, as a tuple; () for a leaf. _op: The symbol of the operation that made this Value, such as "+"; "" for a leaf. _backward: Passes this Value's grad back to its parents. It does nothing for a leaf, or for a Value whose operation has no backward yet. """ def __init__(self, data: float, _children: tuple["Value", ...] = (), _op: str = "") -> None: """Make a Value holding data. Args: data: The number, such as 2 or 0.5; it is kept as a float. _children: The Values this one was made from, its parents; () for a leaf. _op: The symbol of the operation that made it, such as "+"; "" for a leaf. """ # --- gap L07.init --- self.data = float(data) self.grad = 0.0 self._prev = tuple(_children) self._op = _op # --- end --- self._backward: Callable[[], None] = lambda: None def __repr__(self) -> str: """How print shows a Value: its data and its grad. Returns: Text such as Value(2.0, grad=0.0). """ # --- gap L07.repr --- return f"Value({self.data!r}, grad={self.grad!r})" # --- end --- def __add__(self, other: "Value | float") -> "Value": """self + other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the sum, with self and other (as a Value) as its parents and "+" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a + a, ends up with both shares. """ # --- gap L07.add --- other = _as_value(other) out = Value(self.data + other.data, (self, other), "+") # --- end --- # --- gap L09.add_backward --- def _backward(): self.grad += out.grad other.grad += out.grad out._backward = _backward # --- end --- return out def __mul__(self, other: "Value | float") -> "Value": """self * other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the product, with self and other (as a Value) as its parents and "*" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a * a, ends up with both shares. """ # --- gap L07.mul --- other = _as_value(other) out = Value(self.data * other.data, (self, other), "*") # --- end --- # --- gap L09.mul_backward --- def _backward(): self.grad += other.data * out.grad other.grad += self.data * out.grad out._backward = _backward # --- end --- return out # Lesson 07 gives these five forward only; lesson 09 rebuilds them from + and *. def __neg__(self) -> "Value": """-self, as a new Value. Returns: A new Value holding minus this Value's data. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ # --- gap L09.neg --- return self * -1 # --- end --- def __sub__(self, other: "Value | float") -> "Value": """self - other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches both sides through it: per unit of the result's grad, self gets 1 and other, when it is a Value, gets -1. """ # --- gap L09.sub --- return self + (-other) # --- end --- def __rsub__(self, other: "Value | float") -> "Value": """other - self, when the left side is a plain number, as in 2 - a. Args: other: The left side of the minus, a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ # --- gap L09.rsub --- return other + (-self) # --- end --- def __radd__(self, other: "Value | float") -> "Value": """other + self, when the left side is a plain number, as in 2 + a. Args: other: The left side of the plus, a plain number such as 2. Returns: A new Value holding the sum. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets 1 per unit of the result's grad. """ # --- gap L09.radd --- return self + other # --- end --- def __rmul__(self, other: "Value | float") -> "Value": """other * self, when the left side is a plain number, as in 2 * a. Args: other: The left side of the times, a plain number such as 2. Returns: A new Value holding the product. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets other per unit of the result's grad. """ # --- gap L09.rmul --- return self * other # --- end --- # Lesson 07 gives division forward only; lesson 11 rebuilds it from * and **. # --- given in lesson 11 --- def __truediv__(self, other: "Value | float") -> "Value": """self / other, as a new Value. Args: other: A Value, or a plain number such as 2; not 0. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If other is 0. """ return self * other ** -1 def __rtruediv__(self, other: "Value | float") -> "Value": """other / self, when the left side is a plain number, as in 2 / a. Args: other: The left side of the division, a plain number such as 2. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If this Value's data is 0. """ return other * self ** -1 # --- end --- def __pow__(self, exponent: float) -> "Value": """self ** exponent, as a new Value. The exponent is a plain number, never a Value. Args: exponent: An int or a float, such as 2 or -1. Returns: A new Value holding this Value's data to the power exponent, with this Value as its parent and "**" followed by the exponent, such as "**2", as its operation. Raises: TypeError: If exponent is not an int or a float (for example a Value), or if data is negative and exponent is not a whole number. ZeroDivisionError: If this Value's data is 0 and exponent is negative. OverflowError: If the result is too large for a float. """ if not isinstance(exponent, (int, float)): raise TypeError("Value ** needs an int or float exponent, not a Value") # --- gap L11.pow --- out = Value(self.data ** exponent, (self,), f"**{exponent}") # Give out a _backward here that applies the power rule. return out # --- end --- def backward(self) -> None: """Fill in .grad on every Value this one depends on: order the graph so that every node comes after its inputs (with an explicit stack, never recursion), seed this Value's grad with 1.0, then run each node's _backward from this Value back to the leaves.""" # --- gap L10.backward --- order, visited, stack = [], set(), [(self, False)] while stack: node, expanded = stack.pop() if expanded: order.append(node) elif node not in visited: visited.add(node) stack.append((node, True)) stack.extend((child, False) for child in node._prev) self.grad = 1.0 for node in reversed(order): node._backward() # --- end --- def exp(self) -> "Value": """e to the power of this Value. Its local slope is its own output. Returns: A new Value holding e to the power data, with this Value as its parent. Raises: OverflowError: If data is above about 709, where the result is too large for a float. """ out = Value(math.exp(self.data), (self,), "exp") def _backward() -> None: self.grad += out.data * out.grad out._backward = _backward return out def log(self) -> "Value": """The natural log. Its local slope is 1 / data. Returns: A new Value holding the natural log of data, with this Value as its parent. Raises: ValueError: If data is 0 or negative. """ out = Value(math.log(self.data), (self,), "log") def _backward() -> None: self.grad += out.grad / self.data out._backward = _backward return out def tanh(self) -> "Value": """tanh squashes any number into (-1, 1). Returns: A new Value holding tanh of data, between -1 and 1, with this Value as its parent. """ # --- gap L11.tanh --- t = math.tanh(self.data) out = Value(t, (self,), "tanh") def _backward(): self.grad += (1.0 - t) * out.grad # the cheat sheet's formula: check it out._backward = _backward return out # --- end --- def relu(self) -> "Value": """The kink from lesson 05. Its local slope is 1 where data > 0, else 0. Returns: A new Value holding data where data > 0, else 0.0, with this Value as its parent. """ out = Value(self.data if self.data > 0 else 0.0, (self,), "relu") def _backward() -> None: self.grad += (1.0 if self.data > 0 else 0.0) * out.grad out._backward = _backward return outRunning code needs JavaScript. Above is ember/scalar.py as this lab starts.

A check compares $x^{2}$ at x = 1000: backward says 2000, nudging says 2000.0001. Is the backward wrong?

Yes: 0.0001 is a sizeable gap for a derivative

Next to 2000, a gap of 0.0001 is rounding, not a bug.

No: the relative error is about 0.00000005

0.0001 / 2000.0001 is about 0.00000005, far inside any tolerance.

No bug. The same gap next to 0.0002 would be one.

relu has no single slope at x = 0: 0 on the left, 1 on the right. What does a central difference with h = 0.001 measure there?

(relu(0.001) − relu(−0.001)) / 0.002 = 0.001 / 0.002 = 0.5, the average of the two sides. A check exactly at a kink disagrees with either choice, so checks avoid kinks.