Glossary

Every term the course names, in plain words. The number after each term is the lesson that first uses it.

activation: The function a neuron applies to its weighted sum, such as sigmoid, tanh or relu. Between layers it supplies the bends that a stack of straight-line layers lacks.
backpropagation: Getting every partial slope of a loss in one pass backward through the computation graph, multiplying local slopes along the way. It costs about as much as a forward pass, however many knobs there are.
backward pass: The walk from the loss back to the knobs, in which each operation hands its inputs their share of the gradient.
bias: The knob added after the weighted inputs. For a line it is the line's value at an input of 0, where it crosses the vertical axis.
central difference: A slope measured by nudging the input both up and down by the same small amount and dividing the change by twice that amount. It is far more accurate than nudging one way.
chain rule: Along a path of operations, local slopes multiply: the product says how fast the end of the path moves per unit of its start.
computation graph: The web of Values a calculation leaves behind, each Value pointing to the Values it was made from.
decision boundary: The line or curve where a model's answer changes from no to yes.
dot product: Two lists multiplied pair by pair and added up. A neuron's weighted sum is the dot product of its weights and its inputs, plus the bias.
error: The target minus the prediction, for one example. It is positive when the model guessed too low.
example: One input together with the answer it should get, such as a ride's distance and its fare.
finite differences: Measuring slopes by nudging each knob and running the model again. Simple and reliable, and it costs one forward pass per knob.
forward pass: Running the model on its inputs to get its outputs and the loss.
gradient: The list of every partial slope of the loss, one per knob. Stepping against it lowers the loss fastest.
gradient accumulation: Adding each path's share into a Value's gradient instead of overwriting it, so a Value used in several places receives the sum.
gradient check: Comparing the gradient a backward pass computes with one measured by central differences, judged by relative error.
gradient descent: Moving every knob against its partial slope, by the learning rate times that slope, over and over until the loss stops falling.
grid search: Trying every combination of candidate knob values and keeping the one with the lowest loss. Exact, and its cost multiplies with every knob.
hidden layer: A layer between the inputs and the output, whose values the user never sees directly.
layer: A group of neurons that read the same inputs; its outputs are the next layer's inputs.
leaf: A Value made directly rather than by an operation. The knobs and the inputs are leaves.
learning: Setting a model's knobs from examples, instead of being told the rule.
learning rate: How far a gradient-descent step moves per unit of slope. Too small crawls; too large overshoots and can grow without end.
linear function: A weighted sum of inputs plus a bias: a straight line for one input, a flat plane for two. Stacking linear functions gives another one.
local slope: How fast one operation's output moves per unit of one input, with the other inputs held still: 1 for each input of a sum, and the other input for a product.
loss: One number that says how wrong a model is on its examples. Training makes it small.
mean absolute error: The average size of the errors. One miss of 4 costs as much as four misses of 1.
mean squared error: The average of the squared errors. One miss of 4 costs as much as sixteen misses of 1, so big misses dominate.
MLP: A multi-layer perceptron: layers of neurons, each layer feeding the next, with an activation between them.
model: A function with knobs that turns an input into a prediction.
natural log: The function that undoes the exponential, written ln. ember names the method log.
neuron: A weighted sum of inputs plus a bias, passed through an activation.
nonlinearity: A function that is not a straight line, such as relu, placed between layers so that depth adds power.
parameter: A knob of a model, such as a weight or a bias, that training adjusts.
partial slope: How fast the loss changes when one knob is nudged and every other knob is held still. Written with the partial sign, ∂.
perceptron: A neuron whose output is a hard yes or no. Its slope is zero almost everywhere, so gradient descent cannot train it.
prediction: What a model outputs for an input.
recursion limit: How deeply Python lets functions call themselves, about 1,000 by default. Walks over deep graphs keep their own stack instead.
relative error: The gap between two numbers divided by the larger of their sizes, so that big and small numbers are judged alike.
relu: Keeps positive numbers and turns negative ones into 0: the simplest kink.
sigmoid: A smooth step from 0 to 1, equal to 0.5 at zero, near 1 for large inputs and near 0 for very negative ones.
slope: How much a function's output changes per unit change of its input, measured over a tiny nudge.
step function: A function that jumps from 0 to 1 at one point and is flat everywhere else.
tanh: A smooth step from −1 to 1, equal to 0 at zero. The activation of lesson 12's neurons.
target: The answer an example says the model should give.
topological order: An order of a graph's nodes in which every node comes after all the nodes it was made from. Backward runs it in reverse.
training loop: The repeated cycle of forward pass, loss, reset of the gradients, backward pass and update.
two-moons: A small made-up dataset of two interleaved half circles, used to see whether a network can learn a curved boundary.
Value: ember's number that remembers: its data, its gradient, the Values it was made from and the operation that made it.
weight: The knob that multiplies an input. For a line it is the slope.
weighted sum: Each input times its weight, added up.
XOR: Exclusive or: yes when exactly one of two inputs is on. One neuron cannot compute it; two layers with a kink can.
zero_grad: Setting every parameter's gradient back to 0 before the next backward pass, because backward adds to what is already there.