Write a neural network framework, one question at a time.

Grad School: Build a Neural Network Framework from Scratch

You'll build the thing that did this.

Start lesson 1 See the syllabus

Free. No account and nothing to install: every lab runs real Python in your browser.

What you'll build

By the end of the course this is your code: a small convolutional network, defined and trained with ember, the Grad School framework you will have written, down to every backward pass.

from ember import nn, optim
from ember.data import DataLoader, TensorDataset
from ember.nn import functional as F

class SmallCNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.features = nn.Sequential(nn.Conv2d(1, 8, 3), nn.ReLU(), nn.MaxPool2d(2),
                                      nn.Conv2d(8, 16, 3), nn.ReLU(), nn.MaxPool2d(2))
        self.head = nn.Sequential(nn.Flatten(), nn.Linear(16 * 5 * 5, 10))
    def forward(self, x):                      # x: (batch, 1, 28, 28) images
        return self.head(self.features(x))     # (batch, 10) scores
def train(model, images, labels, epochs=1):
    opt = optim.Adam(model.parameters(), lr=1e-3)
    for _ in range(epochs):
        for x, y in DataLoader(TensorDataset(images, labels), batch_size=64, shuffle=True):
            loss = F.cross_entropy(model(x), y)
            opt.zero_grad()
            loss.backward()
            opt.step()

Seven parts get you there, and each one adds a facet to your collection of gems:

How a lesson works

1. Predict. Every lesson opens on a problem and a question, and you commit an answer before anything is explained.
2. Feel it. You run something that fails, or is far too slow, before you see the fix.
3. Derive. The idea is built up from your prediction, in plain words first and then with numbers you can check by hand.
4. Build. You write the piece of ember that the idea needs, in an editor on the page, and hidden tests answer in sentences.
5. Check. One or two last questions carry the idea to a case you haven't seen.

self.grad = other.data * out.grad
other.grad = self.data * out.grad

Who it's for and what math you need

You need: comfortable Python (functions, classes, dunder methods, lists, dicts). You have seen derivatives and matrices, but you may have forgotten them. No NumPy and no machine learning.

You don't need: calculus fluency, linear algebra, NumPy, or any machine-learning vocabulary. Each piece of math arrives when a lesson needs it, as a question first, with a worked number beside every symbol. Lesson 13 teaches the NumPy the course uses.

By the end you can:

• explain why neural networks work: learning as a search for low error, why a nonlinearity plus depth can approximate almost any function, and why gradient descent finds good weights;
• derive the backward pass of any operation from its local derivative and the chain rule, and check it numerically;
• build a reverse-mode autograd engine for tensors, with broadcasting, matrix multiplication, indexing and reductions;
• design a backend and run the same model on a CPU with NumPy or a GPU with CuPy;
• build Linear, Conv2d, pooling, BatchNorm, LayerNorm, Dropout, Embedding, RNN and LSTM layers from nothing, and say why each exists;
• train reliably: choose an initialization, a loss and an optimizer, and handle numerical stability, overfitting, exploding gradients and float16's range;
• read PyTorch's documentation and source as a peer, having built the same abstractions;
• start the planned second course, which builds a transformer on your ember.

How this compares

Other courses and projects cover this ground: Andrej Karpathy's micrograd and his Zero to Hero series, MiniTorch, tinygrad, d2l.ai and fast.ai. Grad School is the path that runs entirely in your browser, with every lab checked by tests, from autograd on single numbers to tensors, convolutional and recurrent networks, and a GPU backend, all in one framework you write yourself.

Questions

Hear when the next part opens

The syllabus lists what is open and what comes next.