Lesson 07 · Part II · Autograd on single numbers

A number that remembers

What must a number record so we can ask where it came from?

~35 min · 2 labs · builds on 06 The bill for nudging

This lesson builds on lessons 01–06. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

A model has 250 knobs. How many forward passes does one nudged gradient cost with numerical_gradient?

251: one where you stand, and one per knob.

A model rounds its output to 0 or 1 before the loss is computed. What does gradient_descent do to its knobs?

It trains them, only more slowly

The measured slope isn't small, it is exactly 0, so every step is exactly 0.

It raises an error, because a jump has no slope

Nothing raises: the nudges land on flat ground on either side of the jump, and the measured slope is 0.

Nothing: small nudges never change the rounded outputs, so every measured slope is 0

The loss changes only when an output flips, and a nudge of a millionth almost never flips one.

Nothing moves. A hard step anywhere in the model flattens the loss, which is why the smooth sigmoid replaced the step.

Going backward

Measuring slopes by nudging costs one forward pass per knob, plus one: 10,001 passes for 10,000 knobs.

The way out is to walk the computation backward, once, from the loss to every knob, asking at each step where a number came from. A plain Python float can't answer that question.

c = a * b gives 6.0. From c alone, can you tell what a and b were?

Yes: 2 and 3

2 · 3 is one of many: 1.5 · 4, 12 · 0.5 and −2 · −3 also give 6.0.

No: 6.0 could come from 2 · 3, 1.5 · 4, 12 · 0.5 and endless other pairs

The product keeps the answer and forgets the question.

Yes, if you also know that c came from a multiplication

Knowing the operation still leaves every pair whose product is 6. You need the inputs too.

A float keeps only its value. The cell below makes c from 2 · 3 and d from 1.5 · 4: both are the same float, 6.0, and nothing in either says which pair made it. To go backward, a number has to record where it came from: the numbers it was made from, and the operation that combined them.

Ask a float where it came from.

a, b = 2.0, 3.0
c = a * b
print(c)
print(type(c).__name__)
print("does c know where it came from?", hasattr(c, "_prev"))
d = 1.5 * 4.0
print(d, "is the same float as c:", d == c)

6.0
float
does c know where it came from? False
6.0 is the same float as c: True

A number with a memory

So make a number that remembers. A Value wraps a float, kept in data. It records the Values it was made from, its parents, in _prev, and the operation that made it in _op. A Value you make yourself, such as a knob or an input, has no parents: it is a leaf. Every + and * builds a new Value, and the Values with their links form a computation graph.

Take w = 0.5, x = 2, b = 1 and a target y = 3. Then $w \cdot x$ = 1.0, the prediction $z = w \cdot x + b$ = 2.0, the miss $z - y$ = −1.0, and the loss $(z - y)^{2}$ = 1.0. Each of those is a Value that knows its parents.

Python turns each operator on a Value into a call to a method with a double-underscore name. You write four of them in the labs below. The rest arrive ready-made and work forward only, until lessons 09 and 11 give them a backward.

you write	Python calls	in this lesson
`Value(2.0)`	`__init__`	lab 1
`print(a)`	`__repr__`	lab 1
`a + b`	`__add__`	lab 2
`a * b`	`__mul__`	lab 2
`2 + a`	`__radd__`	given, forward only
`2 * a`	`__rmul__`	given, forward only
`-a`	`__neg__`	given, forward only
`a - b`	`__sub__`	given, forward only
`2 - a`	`__rsub__`	given, forward only
`a / b`	`__truediv__`	given, forward only
`2 / a`	`__rtruediv__`	given, forward only
`a ** 2`	`__pow__`	given, forward only; the exponent is a plain number

The graph of $L = (w \cdot x + b - y)^{2}$ . Each node shows its value; arrows point from parents to the Value they made. The later steps walk backward, but nothing has computed a gradient yet, so each grad shows zero until lesson 08 fills it in.

node	op	forward value
y	input	3
b	input	1
x	input	2
w	input	0.5
#4	*	1
z	+	2
#6	-	-1
L	**2	1

In that graph, how many parents does the Value z record?

Two: the Value $w \cdot x$ and the Value b. Each operation records only its direct parents.

z = w * x + b could record its parents in _prev two ways: the parent Values themselves, or copies of their numbers, 1.0 and 1.0. Which way lets a walk that starts at the loss reach w?

Either: the numbers are all that going backward needs

From 1.0 and 1.0 the walk has nowhere to go: a number has no _prev, so it stops at z.

Only the parent Values: each parent has its own _prev to follow further back

The Value

w \cdot x

records w and x in its own _prev, so the walk goes from the loss back to z, from z to

w \cdot x

, and from there to w.

Neither: _prev must hold the whole history back to w, not only z's parents

z records only its direct parents. The walk reaches w by following parents of parents, as long as each parent is a Value.

The parent Values. Each one records its own parents, so following _prev from the loss reaches every Value that made it; a copied number is a dead end.

What: write Value.__init__, which keeps data (as a float), grad (0.0 for now), _prev (the parents, as a tuple) and _op (the operation's symbol), and __repr__, which shows a Value as Value(6.0, grad=0.0).

Why: without these four fields there is nothing to walk backward: they are all that going backward will ever read. Lesson 08 fills in grad, and every print of a Value from here on uses your __repr__.

Your ember has a number that can remember.

Besides its number, what must a Value keep so that you could later ask where it came from?
Store four attributes: data (as a float), grad (0.0 for now), _prev (the parents, as a tuple) and _op (the operation's symbol). __repr__ returns a string that shows data and grad.
Nearly the code: four assignments in __init__, with data passed through float(), grad set to 0.0, _prev passed through tuple() and _op kept as given. __repr__ returns an f-string that puts self.data and self.grad in the shape the brief shows.

Value(3).data is the float 3.0.
A fresh Value's grad is 0.0.
A Value made from a number has an empty _prev and an empty _op.
Value(5.0, (a, b), '+') remembers a, b and '+'.
repr(Value(2.0)) is 'Value(2.0, grad=0.0)', and a changed grad shows too.

"""Part II: Value, a number that remembers how it was made (the scalar autograd engine).""" # A type hint that names Value is written in quotes, "Value": this def runs before class Value # exists, and inside the class the name is not ready until the class body ends. Python keeps a # quoted hint as text instead of looking the name up. Value | float means a Value or a plain # number, and tuple["Value", ...] is a tuple of Values of any length. def _as_value(x: "Value | float") -> "Value": """A Value for x: x itself when it already is one, else a new leaf Value holding the number. Args: x: A Value, or a plain number such as 2 or 0.5. Returns: x itself when it is a Value, else a new leaf Value whose data is x as a float. """ if isinstance(x, Value): return x return Value(x) class Value: """A number that remembers the Values it was made from (_prev) and the operation (_op). Attributes: data: The number, as a float. grad: How much the final result (such as a loss) changes for each unit that data grows; 0.0 until a backward pass fills it in. _prev: The Values this one was made from, its parents, as a tuple; () for a leaf. _op: The symbol of the operation that made this Value, such as "+"; "" for a leaf. """ def __init__(self, data: float, _children: tuple["Value", ...] = (), _op: str = "") -> None: """Make a Value holding data. Args: data: The number, such as 2 or 0.5; it is kept as a float. _children: The Values this one was made from, its parents; () for a leaf. _op: The symbol of the operation that made it, such as "+"; "" for a leaf. """ # --- gap L07.init --- raise NotImplementedError("Store data as a float, grad as 0.0, _prev as a tuple of _children, and _op.") # --- end --- def __repr__(self) -> str: """How print shows a Value: its data and its grad. Returns: Text such as Value(2.0, grad=0.0). """ # --- gap L07.repr --- raise NotImplementedError("Return a string like Value(2.0, grad=0.0).") # --- end --- def __add__(self, other: "Value | float") -> "Value": """self + other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the sum, with self and other (as a Value) as its parents and "+" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a + a, ends up with both shares. """ # --- gap L07.add --- raise NotImplementedError("Turn other into a Value with _as_value, then set out to a Value that remembers self, other and '+'.") # --- end --- return out def __mul__(self, other: "Value | float") -> "Value": """self * other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the product, with self and other (as a Value) as its parents and "*" as its operation. From lesson 08 on, a backward pass sends the result's grad back to each parent, times that parent's local slope; from lesson 09 on, an input used twice, as in a * a, ends up with both shares. """ # --- gap L07.mul --- raise NotImplementedError("Turn other into a Value with _as_value, then set out to a Value that remembers self, other and '*'.") # --- end --- return out # Lesson 07 gives these five forward only; lesson 09 rebuilds them from + and *. def __neg__(self) -> "Value": """-self, as a new Value. Returns: A new Value holding minus this Value's data. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ return Value(-self.data, (self,), "neg") def __sub__(self, other: "Value | float") -> "Value": """self - other, as a new Value. Args: other: A Value, or a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches both sides through it: per unit of the result's grad, self gets 1 and other, when it is a Value, gets -1. """ other = _as_value(other) return Value(self.data - other.data, (self, other), "-") def __rsub__(self, other: "Value | float") -> "Value": """other - self, when the left side is a plain number, as in 2 - a. Args: other: The left side of the minus, a plain number such as 2. Returns: A new Value holding the difference. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets -1 per unit of the result's grad. """ other = _as_value(other) return Value(other.data - self.data, (other, self), "-") def __radd__(self, other: "Value | float") -> "Value": """other + self, when the left side is a plain number, as in 2 + a. Args: other: The left side of the plus, a plain number such as 2. Returns: A new Value holding the sum. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets 1 per unit of the result's grad. """ other = _as_value(other) return Value(other.data + self.data, (other, self), "+") def __rmul__(self, other: "Value | float") -> "Value": """other * self, when the left side is a plain number, as in 2 * a. Args: other: The left side of the times, a plain number such as 2. Returns: A new Value holding the product. From lesson 09 on, a backward pass also reaches this Value through it: its grad gets other per unit of the result's grad. """ other = _as_value(other) return Value(other.data * self.data, (other, self), "*") # Lesson 07 gives division forward only; lesson 11 rebuilds it from * and **. def __truediv__(self, other: "Value | float") -> "Value": """self / other, as a new Value. Args: other: A Value, or a plain number such as 2; not 0. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If other is 0. """ other = _as_value(other) return Value(self.data / other.data, (self, other), "/") def __rtruediv__(self, other: "Value | float") -> "Value": """other / self, when the left side is a plain number, as in 2 / a. Args: other: The left side of the division, a plain number such as 2. Returns: A new Value holding the quotient. Raises: ZeroDivisionError: If this Value's data is 0. """ other = _as_value(other) return Value(other.data / self.data, (other, self), "/") def __pow__(self, exponent: float) -> "Value": """self ** exponent, as a new Value. The exponent is a plain number, never a Value. Args: exponent: An int or a float, such as 2 or -1. Returns: A new Value holding this Value's data to the power exponent, with this Value as its parent and "**" followed by the exponent, such as "**2", as its operation. Raises: TypeError: If exponent is not an int or a float (for example a Value), or if data is negative and exponent is not a whole number. ZeroDivisionError: If this Value's data is 0 and exponent is negative. OverflowError: If the result is too large for a float. """ if not isinstance(exponent, (int, float)): raise TypeError("Value ** needs an int or float exponent, not a Value") # Forward only until lesson 11 gives ** its backward. return Value(self.data ** exponent, (self,), f"**{exponent}")Running code needs JavaScript. Above is ember/scalar.py as this lab starts.

What: make + and * on Values return a new Value that holds the result, with the two inputs as its parents and '+' or '*' as its operation, wrapping a plain number with the helper _as_value first.

Why: without these records, going backward has no way back. Every later lesson reads them: lesson 08 walks them backward, lesson 10 puts them in order, and the graph above is drawn from them.

Your ember's numbers remember how they were made.

Two cases for other: it is already a Value, or it is a plain number like 2.0. What must happen to a plain number before it can be recorded as a parent?
Wrap it with the helper above the class: other = _as_value(other). Then out is a new Value holding the result, with (self, other) as its parents and '+' or '*' as its operation.
Nearly the code: after other = _as_value(other), set out to a Value made from self.data * other.data, with the parents (self, other) and the symbol '*'. For +, use the sum and '+'.

Value(2) + Value(3) has data 5.0.
a + b remembers a, b and '+'.
a * b has data 6.0 and remembers a, b and '*'.
In a + 2 and a * 3 the number is wrapped as a Value and remembered.
In z = w * x + b, z remembers the product, which remembers w and x.
a + a works and remembers a.

Build the loss above with your Values. Running it redraws the graph above from them.

import hx
from ember.scalar import Value

w, x, b, y = Value(0.5), Value(2.0), Value(1.0), Value(3.0)
for v, name in [(w, "w"), (x, "x"), (b, "b"), (y, "y")]:
    v.label = name
z = w * x + b
z.label = "z"
L = (z - y) ** 2
L.label = "L"
print(f"z = {z.data}, made by {z._op!r} from {sorted(p.data for p in z._prev)}")
print(f"L = {L.data}")
hx.show_graph(L)

z = 2.0, made by '+' from [1.0, 1.0]
L = 1.0

You make a and b as Values and write d = (a + b) * (a + b). How many new Values do the operations create?

2: the sum is made once and used twice

Writing a + b twice runs + twice, so there are two sum Values with the same data, and one product.

3: two sums and one product

Each + and each * builds a new Value, even when the expression repeats.

1: the whole formula is stored as one Value

Every operation makes its own Value; that is what lets the graph be walked one step at a time.

Three. Two different Values can hold the same number, which is not the same thing as one Value used twice; lesson 09 depends on the difference.

Where this goes

Your Values remember how they were made. Lesson 08 walks that record backward, so that each knob learns how much of the loss it owes.

When you finish this lesson

Your ember can now: record how every number was made, and draw it as a graph

It already could:

measure every slope of a loss at once, and count the cost
compute XOR with two layers and a kink
train a neuron that decides AND and OR

Next Who owes the gradient? · Each op knows only its local slope. Is that enough? · ~40 min

Continue

Previous 06 The bill for nudging Next 08 Who owes the gradient? Each op knows only its local slope. Is that enough?

Report a problem with this lesson