Lesson 04 · Part I · Why learning works at all

The simplest decision

What is the simplest thing that makes a decision?

~40 min · 2 labs · builds on 03 Downhill in the dark

This lesson builds on lessons 01–03. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

A knob sits at w = 3 and its partial slope there is 4. One gradient-descent step with lr = 0.1: where does w land?

3 − 0.1 · 4 = 2.6: against the slope.

A program learns to predict a plant's height from its days of sunlight. After training, its line has w = 0.8. Who chose 0.8?

The programmer, who knew the growth rule and typed it in

If the rule were known, there would be nothing to learn and no training to run.

One plant: the program copied the growth rate of the plant most like the new one

No single example is copied: 0.8 is fitted to all the plants at once.

Nobody: it is the value that made the line agree best with the measured plants

The programmer chose the shape, a line, and how to score it; the measurements chose 0.8.

The data chose it, through training. That is the whole idea of lesson 01, on a new surface.

Yes or no

gradient_descent tunes the knobs of any function by feeling the slope of its loss, one nudge per knob.

So far the function drew a line through numbers. Many questions want a yes or a no instead: should a gate open when both of its sensors fire? A neuron answers questions like that. It multiplies each input by a weight, adds up the products and a bias, and says yes when the total is above 0.

A neuron answers yes when $w_1 x_1 + w_2 x_2 + b$ is above 0, for inputs $x_1$ and $x_2$ that are each 0 or 1. Which of these rules can no setting of $w_1$ , $w_2$ and b produce?

AND: yes only when both inputs are 1

w_1 = w_2 = 2

and b = −3 does it: only 2 + 2 − 3 is above 0. One straight line cuts the corner (1, 1) off from the other three.

XOR: yes when exactly one input is 1

The yes corners, (0, 1) and (1, 0), sit on one diagonal and the no corners on the other. No straight line puts both yeses on one side and both nos on the other.

None: with the right numbers one neuron can produce any of these rules

One neuron's yes-region is everything on one side of a straight line, and XOR's two yes corners can't be fenced off by one line.

XOR. A single neuron splits its inputs with one straight line. AND and OR each need one cut; XOR needs two, and lesson 05 makes them.

A neuron with that hard answer, 1 when the total is above 0 and 0 otherwise, is called a perceptron. Hand one to gradient_descent and ask it to learn AND.

A step neuron gets AND wrong on two of four inputs. Measure its slopes, then let gradient_descent try for 100 steps.

from ember.basics import numerical_derivative, gradient_descent, mse

AND = [((0, 0), 0), ((0, 1), 0), ((1, 0), 0), ((1, 1), 1)]


def step(z):
    return 1.0 if z > 0 else 0.0


def loss(knobs):
    w1, w2, b = knobs
    return mse([t for _, t in AND], [step(w1 * x1 + w2 * x2 + b) for (x1, x2), _ in AND])


knobs = [0.3, -0.2, 0.1]
print(f"loss {loss(knobs):.2f}")
for i, name in enumerate(["w1", "w2", "b"]):
    def along(v, i=i):
        moved = list(knobs)
        moved[i] = v
        return loss(moved)
    print(f"measured slope for {name}: {numerical_derivative(along, knobs[i]):.2f}")
after, _ = gradient_descent(loss, knobs, 1.0, 100)
print(f"after 100 steps: knobs {[round(k, 2) for k in after]}, loss {loss(after):.2f}")

loss 0.50
measured slope for w1: 0.00
measured slope for w2: 0.00
measured slope for b: 0.00
after 100 steps: knobs [0.3, -0.2, 0.1], loss 0.50

The loss is 0.50, all three measured slopes are 0.00, and after 100 steps the knobs have not moved. A nudge of a millionth never flips an answer, so the loss never changes, so there is no slope to follow.

A smooth step

Multiplying two lists pair by pair and adding up the products is called a dot product. The weights (2, −1) and the inputs (1, 0) give 2 · 1 + (−1) · 0 = 2. A neuron's weighted sum is the dot product of its weights and its inputs.

The neuron's total is a weighted sum of its inputs plus the bias, $z = \sum_{i} w_i x_i + b$ . For x = (1, 0), w = (2, −1) and b = −0.5: z = 2 · 1 + (−1) · 0 − 0.5 = 1.5.

The step function is flat everywhere except at its jump, so its measured slope is 0 almost everywhere, and gradient descent has nothing to follow. The neuron needs a step with a slope: one that rises smoothly from 0 to 1. Its ingredient is a new number, e, raised to a power.

The number e is about 2.718, and $e^{0} = 1$ while $e^{-1} \approx 0.368$ : $e^{x}$ shrinks towards 0 as x goes negative and grows fast as x goes positive.

The smooth step is $\mathrm{sigmoid}(z) = \frac{1}{1 + e^{-z}}$ which at z = 1.5 is 1 / (1 + 0.2231) = 0.8176, because $e^{-1.5} \approx 0.2231$ .

sigmoid is 0.5 at z = 0, close to 1 for large z and close to 0 for very negative z. Read an output above 0.5 as yes. Every small change of z now changes the output a little, so the loss has a slope to feel. The function a neuron applies to its total, here sigmoid, is called its activation.

Using $e^{-1} \approx 0.368$ , what is sigmoid(1)?

1 / (1 + 0.368) = 1 / 1.368, about 0.731: a fairly confident yes.

What: write sigmoid(z), and neuron(xs, ws, b), which returns sigmoid of the weighted sum of the inputs plus b.

Why: without a smooth step the neuron has no slope, and nothing can train it. train_neuron in the next lab tunes this neuron, lesson 12's Neuron is the same shape built from Values, and lesson 23 compares sigmoid with other kinks.

Your ember has a neuron.

sigmoid needs e raised to the power -z. Which function in the math module computes e to a power?
sigmoid is the page's formula, with math.exp(-z) for e to the power -z. neuron pairs each weight with its input, adds up the products, adds b, and passes the total through sigmoid.
Nearly the code: sigmoid returns 1 / (1 + math.exp(-z)); neuron returns sigmoid(sum(w * x for w, x in zip(ws, xs)) + ...)

sigmoid(0) is exactly 0.5, halfway between off and on.
sigmoid(2) is about 0.8808 and sigmoid(-2) about 0.1192.
sigmoid(-z) is 1 - sigmoid(z).
Even at z = 30 and z = -30 the output is inside (0, 1).
neuron(xs, ws, b) is sigmoid(w . x + b).
With all-zero inputs the neuron's output is sigmoid(b).

"""Part I: learning with plain Python numbers and lists (no NumPy, no framework yet).""" import math from collections.abc import Callable, Sequence # The type hints after a parameter (x: float) and after -> say what a function takes and # returns. Python does not check them. # Sequence[float] is a list or a tuple of numbers, and Callable[[float], float] is a function # that takes one number and returns one. def predict_line(x: float, w: float, b: float) -> float: """The line's prediction at x: the slope w times x, plus the intercept b. Args: x: The input, such as a ride's distance in km. w: The slope: how much the prediction grows for each unit of x. b: The intercept: the prediction at x = 0. Returns: The prediction for x, a number in the units of the targets. """ # --- gap L01.predict_line --- return w * x + b # --- end --- def mse(ys: Sequence[float], preds: Sequence[float]) -> float: """Mean squared error: the average of (y - pred) squared over the pairs. Args: ys: The targets; at least one. preds: The predictions, one per target (same length as ys): preds[i] goes with ys[i]. Returns: The loss: 0.0 when every prediction equals its target, and never negative. Raises: ZeroDivisionError: If ys is empty. """ # --- gap L02.mse --- return sum((y - p) ** 2 for y, p in zip(ys, preds)) / len(ys) # --- end --- def mae(ys: Sequence[float], preds: Sequence[float]) -> float: """Mean absolute error: the average of |y - pred| over the pairs. Args: ys: The targets; at least one. preds: The predictions, one per target (same length as ys): preds[i] goes with ys[i]. Returns: The loss: 0.0 when every prediction equals its target, and never negative. Raises: ZeroDivisionError: If ys is empty. """ # --- gap L02.mae --- return sum(abs(y - p) for y, p in zip(ys, preds)) / len(ys) # --- end --- def grid_search( xs: Sequence[float], ys: Sequence[float], ws: Sequence[float], bs: Sequence[float], loss: Callable[[Sequence[float], Sequence[float]], float], ) -> tuple[float, float]: """Try the line of every pair (w, b) from ws and bs and return the pair (w, b) whose predictions get the lowest loss(ys, preds). On a tie the pair tried first wins (ws is the outer loop). Args: xs: The inputs. ys: The targets, one per input (same length as xs). ws: The slopes to try; at least one. bs: The intercepts to try; at least one. loss: A score for one line, such as mse or mae: loss(ys, preds) takes the targets and that line's predictions for xs, in order, and returns a number; lower is better. Returns: The best pair (w, b), with w taken from ws and b from bs. """ # --- gap L02.grid_search --- best, best_loss = None, None for w in ws: for b in bs: value = loss(ys, [predict_line(x, w, b) for x in xs]) if best_loss is None or value < best_loss: best, best_loss = (w, b), value return best # --- end --- def numerical_derivative(f: Callable[[float], float], x: float, h: float = 1e-6) -> float: """The slope of f at x, measured by nudging x forward by h: (f(x + h) - f(x)) / h. Args: f: A function that takes one number and returns one number. x: Where to measure the slope. h: The size of the nudge: small, and not 0. Returns: The measured slope: about how much f changes for each unit that x grows, near x. Raises: ZeroDivisionError: If h is 0. """ # --- gap L03.numerical_derivative --- return (f(x + h) - f(x)) / h # --- end --- def gradient_descent( f: Callable[[list[float]], float], params: list[float], lr: float, steps: int, h: float = 1e-6, ) -> tuple[list[float], list[float]]: """Walk downhill on f, a function of a list of knobs. Each step measures every knob's partial slope with numerical_derivative (nudge that knob, hold the others still), and only then moves every knob by -lr times its slope. Returns (the final knobs as a new list, the losses), where losses holds f before the first step and after every step (steps + 1 numbers). Args: f: The loss: takes a list of knobs, as many as params holds, and returns a number. params: The knobs to start from. The caller's list is not changed. lr: The learning rate, usually small and positive, such as 0.01. steps: How many steps to take; 0 or more. h: The nudge numerical_derivative uses to measure each slope. Returns: (knobs, losses): knobs is a new list with one number per knob of params, and losses holds steps + 1 numbers, the first being f(params). """ # --- gap L03.gradient_descent --- params = list(params) losses = [f(params)] for _ in range(steps): slopes = [] for i in range(len(params)): def along(v, i=i): return f(params[:i] + [v] + params[i + 1:]) slopes.append(numerical_derivative(along, params[i], h)) params = [p - lr * s for p, s in zip(params, slopes)] losses.append(f(params)) return params, losses # --- end --- def sigmoid(z: float) -> float: """The smooth step 1 / (1 + e^-z): near 0 for very negative z, 0.5 at 0, near 1 for large z. Args: z: Any number, such as a neuron's weighted sum plus its bias. Returns: A number between 0 and 1, rising as z rises. Above 0.5 reads as yes. Raises: OverflowError: If z is below about -709, where e^-z is too large for a float. """ # --- gap L04.sigmoid --- raise NotImplementedError("Return 1 / (1 + e^-z); math.exp is imported for you.") # --- end --- def neuron(xs: Sequence[float], ws: Sequence[float], b: float) -> float: """One neuron: the sigmoid of the weighted sum of the inputs plus the bias. Args: xs: The inputs. ws: The weights, one per input (same length as xs): ws[i] goes with xs[i]. b: The bias. Returns: The neuron's output, between 0 and 1. Above 0.5 reads as yes. """ # --- gap L04.neuron --- raise NotImplementedError("Squash the weighted sum of xs plus b with sigmoid.") # --- end --- def train_neuron( data: Sequence[tuple[Sequence[float], float]], lr: float = 1.0, steps: int = 1000, ) -> tuple[list[float], float]: """Fit one neuron to data, a list of (inputs, target) pairs with targets 0 or 1: run gradient_descent on the mse of the neuron's outputs, over the knobs [weights..., bias], starting from all zeros. Returns (weights as a list, bias). Args: data: The examples; at least one. Every example has the same number of inputs. lr: The learning rate handed to gradient_descent. steps: How many gradient_descent steps to take; 0 or more. Returns: (ws, b): ws holds one weight per input of an example, and b is the bias. With steps = 0 every one of them is 0.0. """ # --- gap L04.train_neuron --- raise NotImplementedError("Hand gradient_descent a loss over the knobs [weights..., bias], starting from zeros.") # --- end ---Running code needs JavaScript. Above is ember/basics.py as this lab starts.

What: write train_neuron(data, lr, steps): data is a list of (inputs, target) pairs; fit the neuron's weights and bias with gradient_descent on the mean squared error of its outputs, and return (ws, b).

Why: without it a neuron's weights are numbers you pick by hand. It is the first time your ember learns a decision from examples, and lesson 05 needs its failure on XOR.

Your ember learns AND and OR from four examples each.

gradient_descent tunes one flat list of knobs. How do you pack the weights and the bias into one list, and unpack them inside the loss?
Write loss(knobs): ws = knobs[:-1] and b = knobs[-1]; return the mse of the targets against neuron(x, ws, b) for every example. Start from zeros: one per input, plus one for the bias.
Nearly the code: knobs, _ = gradient_descent(loss, [0.0] * (len(data[0][0]) + 1), lr, steps); then return knobs[:-1], knobs[-1].

With steps=0 the knobs stay at zero: ([0, 0], 0).
One step at lr=1 on AND leaves the weights at 0 and moves the bias to -0.125.
Trained with the defaults, one neuron gets all four AND cases right.
Trained with the defaults, one neuron gets all four OR cases right.
No single neuron gets all four XOR cases right, so training must leave one wrong.

Train on AND, OR and XOR, and draw the AND neuron's boundary.

import hx
from ember.basics import neuron, train_neuron

AND = [((0, 0), 0), ((0, 1), 0), ((1, 0), 0), ((1, 1), 1)]
OR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 1)]
XOR = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]

trained = {}
for name, data in [("AND", AND), ("OR", OR), ("XOR", XOR)]:
    ws, b = train_neuron(data, 10.0, 300)
    trained[name] = (ws, b)
    print(f"{name:3}: " + "  ".join(f"{x} -> {neuron(x, ws, b):.2f}" for x, _ in data))

ws, b = trained["AND"]
grid = {"x0": -0.5, "x1": 1.5, "y0": -0.5, "y1": 1.5, "nx": 21, "ny": 21}
values = [[neuron((-0.5 + 0.1 * i, -0.5 + 0.1 * j), ws, b) for i in range(21)] for j in range(21)]
points = [{"x": x1, "y": x2, "c": t} for (x1, x2), t in AND]
hx.plot("boundary", {"grid": grid, "values": values, "points": points})

AND: (0, 0) -> 0.00  (0, 1) -> 0.04  (1, 0) -> 0.04  (1, 1) -> 0.95
OR : (0, 0) -> 0.04  (0, 1) -> 0.97  (1, 0) -> 0.97  (1, 1) -> 1.00
XOR: (0, 0) -> 0.50  (0, 1) -> 0.50  (1, 0) -> 0.50  (1, 1) -> 0.50

The trained AND neuron's output over the plane: one straight decision boundary, with (1, 1) alone on the yes side.

class	points
0	3
1	1
21×21 grid, 27% of it on the class 1 side

AND and OR come out right. XOR gets 0.50 for every input: the best a single straight cut can do is to say "maybe" everywhere.

A trained neuron has $w_1 = 3$ , $w_2 = 3$ and b = −4. Which inputs does it say yes to (output above 0.5)?

(0, 1), (1, 0) and (1, 1): each weight of 3 says yes on its own

Only if you leave out the bias. With it, one input on gives 3 − 4 = −1, below 0.

Only (1, 1): it computes AND

z is −4, −1, −1 and 2 for the four inputs, and above 0 means yes.

All four: sigmoid's output is always positive

sigmoid(−4) ≈ 0.018 is positive but far below 0.5, and the neuron's yes means above 0.5.

AND. The weights say how much each input counts, and the bias sets how much evidence the neuron needs before it says yes.

Where this goes

One neuron draws one straight boundary, and XOR needs two. Lesson 05 stacks neurons, and finds that stacking alone is not enough without a kink.

When you finish this lesson

Your ember can now: train a neuron that decides AND and OR

It already could:

fit a line by walking downhill on its loss
score any line with one number, and search a grid for the best one
turn two knobs, w and b, into a prediction

Next Bending space · Why stack layers, and why must there be a kink? · ~35 min

Continue

Previous 03 Downhill in the dark Next 05 Bending space Why stack layers, and why must there be a kink?

Report a problem with this lesson