Lesson 02 · Part I · Why learning works at all

How wrong is wrong?

Why square the error instead of adding it up?

~35 min · 2 labs · builds on 01 A function with knobs

This lesson builds on lesson 01. New here? Start at lesson 01, or carry on and load the finished pieces with one click.

A line has w = 1.5 and b = 2. What does predict_line(4, 1.5, 2) return?

1.5 · 4 + 2 = 8. The slope does its work once per unit of x; b is added once.

A line passes through (0, 3) and (2, 7). What is its w?

From x = 0 to x = 2 the line climbs 4, so it climbs 2 per unit: w = 2. At x = 0 it reads 3, so b = 3.

One number for a whole line

A line with two knobs, w and b, turns a distance into a fare. By eye you can tell a bad line from a good one, but not a good one from a slightly better one.

Take two lines fitted by eye to five taxi receipts. Ana's line, w = 2.4 and b = 3.4, misses the receipts by 0.24 on average and never by more than 0.3. Ben's line, w = 2.6 and b = 2.6, misses by 0.2 on average, but by 0.5 once. Both look good, so your eye can't choose between them.

A program needs one number per line that says how wrong the line is. That number is called a loss, and the smaller it is, the better the line.

Four points: (1, 3), (2, 5), (3, 7), (4, 9). Line A is $y = 2x + 1$ , which goes through all four. Line B is the flat line $y = 6$ . Score each line by the average of its errors, where an error is target minus prediction. Which line gets the smaller score?

A: its errors are all 0, and B misses every point, so B's average can't be 0

B's errors are 3 − 6, 5 − 6, 7 − 6 and 9 − 6: −3, −1, +1 and +3. They add up to 0. An average of zero says the misses balance, not that there are none.

A: B scores 2, because its errors are 3, 1, 1 and 3

2 is the average size of B's misses. With the signs kept, as the rule says, −3 and +3 cancel, and so do −1 and +1.

Neither: both score 0

A's errors are 0, 0, 0 and 0. B's are −3, −1, +1 and +3. Both average 0, so this score can't tell a perfect line from a flat one.

Both lines score 0. Errors above the line pay for errors below it, so a score that keeps the signs rewards any line that runs through the middle of the points.

The cell below hands this score to a program. Of its 441 lines, the flat line w = 0.0, b = 6.0 wins: line B, whose errors, −3, −1, +1 and +3, add up to exactly 0. The line through all four points also totals 0, but the flat line is tried first, and this score can't tell them apart.

Here is a program that trusts the signed errors. It tries 441 lines on the four points (1, 3), (2, 5), (3, 7) and (4, 9), adds up each line's errors with their signs, and keeps the line whose total is closest to 0. Trying every pair on a grid like this is called a grid search.

Try 21 slopes and 21 intercepts, 441 lines, and keep the one whose signed errors add up closest to 0.

from ember.basics import predict_line

xs = [1, 2, 3, 4]
ys = [3, 5, 7, 9]
ws = [i * 0.25 for i in range(21)]        # slopes 0, 0.25, ..., 5
bs = [i * 0.5 for i in range(21)]         # intercepts 0, 0.5, ..., 10


def summed_error(ys, preds):
    """Add up the errors with their signs; the closer the total is to 0, the better the score."""
    return abs(sum(y - p for y, p in zip(ys, preds)))


best = None
for w in ws:
    for b in bs:
        score = summed_error(ys, [predict_line(x, w, b) for x in xs])
        if best is None or score < best[0]:
            best = (score, w, b)
score, w, b = best
print(f"tried {len(ws) * len(bs)} lines; the winner's errors add up to {score}, with w = {w}, b = {b}")
for x, y in zip(xs, ys):
    guess = predict_line(x, w, b)
    print(f"x = {x}: target {y}, this line says {guess}, error {y - guess:+}")

tried 441 lines; the winner's errors add up to 0.0, with w = 0.0, b = 6.0
x = 1: target 3, this line says 6.0, error -3.0
x = 2: target 5, this line says 6.0, error -1.0
x = 3: target 7, this line says 6.0, error +1.0
x = 4: target 9, this line says 6.0, error +3.0

Make every miss count

There are two ways to stop misses cancelling. Count each miss by its size, whatever its sign, or by its square, which is never negative either. Either way the loss adds up one term for each example, and a new sign writes that adding up. In the formulas below, n is the number of examples, and $x_i$ and $y_i$ are the input and the target of example number i.

The sign $\sum_{i=1}^{n}$ means "add up the term for each example, from i = 1 to n". For the four points, $\sum_{i=1}^{4} x_i = 1 + 2 + 3 + 4 = 10$ .

The mean squared error squares each miss and averages: $L = \frac{1}{n}\sum_{i=1}^{n} (y_i - (w x_i + b))^2$ For the flat line B that is $((-3)^{2} + (-1)^{2} + 1^{2} + 3^{2}) / 4$ = (9 + 1 + 1 + 9) / 4 = 5.

The mean absolute error averages the sizes instead: $L = \frac{1}{n}\sum_{i=1}^{n} \lvert y_i - (w x_i + b) \rvert$ For B that is $(\lvert -3 \rvert + \lvert -1 \rvert + \lvert 1 \rvert + \lvert 3 \rvert) / 4$ = (3 + 1 + 1 + 3) / 4 = 2.

Why keep the square when the size already stops the cancelling? The square charges more for one big miss than for several small ones: one miss of 4 costs 16, while four misses of 1 cost 4 in all. It is also smooth where a miss passes through 0, where the size has a sharp corner, and the walk downhill in lesson 03 needs that smoothness.

The line $y = 2x + 2$ misses each of the four points by the same amount. What is its mean squared error?

Its errors are −1, −1, −1 and −1. The squares are 1, 1, 1 and 1, which sum to 4, and 4 / 4 = 1. Its mean absolute error is 1 too.

What: write mse(ys, preds) and mae(ys, preds): the mean of the squared errors, and the mean of their sizes.

Why: without a loss, nothing can say which of two lines is better. Every training loop in the course lowers one: lesson 03's gradient descent walks downhill on mse, and lesson 25's cross-entropy replaces it for classes.

Your ember can score a line with one number.

An error is one target minus its prediction. Walk the two lists side by side: which built-in pairs them up?
zip(ys, preds) gives (y, p) pairs. Square each difference for mse, or take its abs for mae; add them all up with sum, and divide by how many there are.
Nearly the code: return sum((y - p) ** 2 for y, p in zip(ys, preds)) / len(ys). mae is the same line with abs(y - p).

mse is the average of the squared differences.
mae is the average of the absolute differences.
A miss above and a miss below both cost something.
Predictions equal to the targets score exactly 0.
Squaring punishes one big miss more than the absolute value does.

"""Part I: learning with plain Python numbers and lists (no NumPy, no framework yet).""" from collections.abc import Callable, Sequence # The type hints after a parameter (x: float) and after -> say what a function takes and # returns. Python does not check them. # Sequence[float] is a list or a tuple of numbers, and Callable[[float], float] is a function # that takes one number and returns one. def predict_line(x: float, w: float, b: float) -> float: """The line's prediction at x: the slope w times x, plus the intercept b. Args: x: The input, such as a ride's distance in km. w: The slope: how much the prediction grows for each unit of x. b: The intercept: the prediction at x = 0. Returns: The prediction for x, a number in the units of the targets. """ # --- gap L01.predict_line --- return w * x + b # --- end --- def mse(ys: Sequence[float], preds: Sequence[float]) -> float: """Mean squared error: the average of (y - pred) squared over the pairs. Args: ys: The targets; at least one. preds: The predictions, one per target (same length as ys): preds[i] goes with ys[i]. Returns: The loss: 0.0 when every prediction equals its target, and never negative. Raises: ZeroDivisionError: If ys is empty. """ # --- gap L02.mse --- raise NotImplementedError("Average the squared differences between ys and preds.") # --- end --- def mae(ys: Sequence[float], preds: Sequence[float]) -> float: """Mean absolute error: the average of |y - pred| over the pairs. Args: ys: The targets; at least one. preds: The predictions, one per target (same length as ys): preds[i] goes with ys[i]. Returns: The loss: 0.0 when every prediction equals its target, and never negative. Raises: ZeroDivisionError: If ys is empty. """ # --- gap L02.mae --- raise NotImplementedError("Average the absolute differences between ys and preds.") # --- end --- def grid_search( xs: Sequence[float], ys: Sequence[float], ws: Sequence[float], bs: Sequence[float], loss: Callable[[Sequence[float], Sequence[float]], float], ) -> tuple[float, float]: """Try the line of every pair (w, b) from ws and bs and return the pair (w, b) whose predictions get the lowest loss(ys, preds). On a tie the pair tried first wins (ws is the outer loop). Args: xs: The inputs. ys: The targets, one per input (same length as xs). ws: The slopes to try; at least one. bs: The intercepts to try; at least one. loss: A score for one line, such as mse or mae: loss(ys, preds) takes the targets and that line's predictions for xs, in order, and returns a number; lower is better. Returns: The best pair (w, b), with w taken from ws and b from bs. """ # --- gap L02.grid_search --- raise NotImplementedError("Score the line of every (w, b) pair with loss and return the best pair.") # --- end ---Running code needs JavaScript. Above is ember/basics.py as this lab starts.

What: write grid_search(xs, ys, ws, bs, loss): try every (w, b) pair from the two lists and return the pair whose predictions give the lowest loss.

Why: without it there is nothing to measure a faster method against. It always finds the best line on its grid, so it is the honest baseline. Lesson 03 uses its cost, every pair every time, to show why walking downhill wins.

Your ember can pick the best of a grid of lines.

Two knobs, each with a list of candidates: how do you visit every pair? Keep the best pair you have seen so far, and its score.
A for loop over ws with a for loop over bs inside it. For each pair, build the predictions with predict_line, score them with loss(ys, preds), and remember the pair when its score is lower than the best so far.
Nearly the code: start with best = None; inside the loops, if best is None or score < best[0]: best = (score, w, b); at the end, return best[1], best[2].

With mse, grid_search picks the (w, b) whose line passes through y = 2x + 1.
grid_search ranks the lines with the loss function it receives, not a fixed one.
When every line scores the same, grid_search returns the first pair it tried.

Run your grid_search on the four points three times: with the summed signed error, with mse and with mae.

from ember.basics import predict_line, mse, mae, grid_search

xs = [1, 2, 3, 4]
ys = [3, 5, 7, 9]
ws = [i * 0.25 for i in range(21)]
bs = [i * 0.5 for i in range(21)]


def summed_error(ys, preds):
    return abs(sum(y - p for y, p in zip(ys, preds)))


for name, loss in [("summed error", summed_error), ("mse", mse), ("mae", mae)]:
    w, b = grid_search(xs, ys, ws, bs, loss)
    preds = [predict_line(x, w, b) for x in xs]
    print(f"grid_search with {name:12}: w = {w}, b = {b}; its mse is {mse(ys, preds):.2f}")

grid_search with summed error: w = 0.0, b = 6.0; its mse is 5.00
grid_search with mse         : w = 2.0, b = 1.0; its mse is 0.00
grid_search with mae         : w = 2.0, b = 1.0; its mse is 0.00

Handed the summed signed error, your grid_search keeps the flat line again, with an MSE of 5.00. Handed mse or mae, it finds w = 2.0 and b = 1.0 among the same 441 lines: the line through all four points.

Model P misses four examples by 0, 0, 1 and 5. Model Q misses them by 2, 2, 1 and 1. Which model does each loss prefer?

Both prefer Q: the two losses always rank models the same way

MAE scores both at exactly 1.5: P's misses and Q's add up to the same 6. Only the square charges extra for P's miss of 5.

MSE prefers Q; MAE can't choose between them

MSE: P scores 26 / 4 = 6.5 and Q scores 10 / 4 = 2.5. MAE: both score 6 / 4 = 1.5. Squaring charges extra for P's one big miss.

Both prefer P: two of its four answers are exact

Exact answers count for nothing extra in either loss; what counts is the misses, and P's miss of 5 is large.

MSE picks Q and MAE calls it a tie. Which one you want depends on whether one large miss is worse than several small ones.

The two losses split on Ana's and Ben's lines from the start of this lesson too. Ana's line misses the receipts by 0.2, 0.3, 0.2, 0.2 and 0.3; Ben's by 0.4, 0.1, 0, 0 and 0.5. MAE prefers Ben's, 0.20 against Ana's 0.24. MSE prefers Ana's, 0.060 against Ben's 0.084: Ben's one miss of 0.5 costs 0.25 once squared, while Ana's largest costs 0.09.

grid_search tried 21 values for each of 2 knobs: 441 lines. With 3 knobs and 21 values each, how many tries?

21 · 21 · 21 = 9261. With 10 knobs it would be $21^{10}$ = 16,679,880,978,201 tries, for a model far smaller than any real one.

Where this goes

Trying every line is exact and hopeless: its cost multiplies with every knob. Lesson 03 finds the best line without trying them all, by feeling which way is downhill and stepping.

When you finish this lesson

Your ember can now: score any line with one number, and search a grid for the best one

It already could:

turn two knobs, w and b, into a prediction

Next Downhill in the dark · You can only feel the slope underfoot. Now what? · ~40 min

Continue

Previous 01 A function with knobs Next 03 Downhill in the dark You can only feel the slope underfoot. Now what?

Report a problem with this lesson