From Framework User to Framework Understander — A 4-Month Journey Through Gradient Descent, Transformers, and the Physics of History
"Every timeline you've ever seen is a lie of omission. It shows you WHEN things happened. It never shows you WHY. ChronoWeave is our attempt to fix that — and along the way, you're going to learn what actually happens inside a neural network when you call
.backward()."
How to Use This Guide
This is not a tutorial you skim. It's a companion for a 3–5 month build. Read one section, do the exercise, hit the wall, climb over it, then come back. Every chapter follows the same rhythm:
- The Concept — the idea, explained with an analogy before a single line of code
- The Code Challenge — a skeleton you fill in, never a finished solution
- The Aha Moment — what you should see when it clicks
- The Extension — how this piece bolts onto the larger machine
- Resources — exact docs, not "just Google it"
And threaded throughout:
- 🧭 The Mentor Says — the advice I wish someone had given me
- 🕳️ The Rabbit Hole — optional, for when curiosity gets the better of you
- 😤 The Struggle — the wall you will hit, named in advance
- ✅ Checkpoint — stop and verify before you go further
- 💡 The Innovation — why this piece of the project is worth talking about in an interview
Let's begin.
CHAPTER 0: The Grand Vision — Seeing the Complete Picture
The Problem: Why Historical Timelines Are Broken
Open any history textbook's timeline. You'll see a horizontal line, some dots, some dates. 1914. 1917. 1929. 1939. It tells you when. It is silent on why.
The assassination of Archduke Franz Ferdinand didn't cause World War I in a vacuum — it was the spark that landed on a decade of alliance treaties, arms races, and colonial tension that had already turned the continent into dry kindling. A flat timeline shows you the spark. It hides the kindling.
What historians actually think in is a causal graph: event A enabled event B, which, combined with condition C, triggered event D. That graph is tangled, non-linear, and multi-dimensional — which is exactly why nobody draws it by hand. It's too much cognitive load.
ChronoWeave's bet: if a machine can extract the causal graph from raw text, and then lay it out so that causally-connected events cluster together and pull each other into readable arrangement, you get something a static timeline can never give you — a map you can explore, drag, and reorganize, where the layout itself is meaningful. Events that influence each other stay near each other. That's not a UI nicety. That's the entire value proposition.
The Solution: ChronoWeave's Architecture
Here's the complete system, before we write a single line of code:
┌─────────────────────────────────────────────────────────────────────┐
│ THE USER'S BROWSER │
│ ┌──────────────┐ ┌────────────────────────────────────┐ │
│ │ Text Input │──────▶│ React + D3.js Canvas │ │
│ │ "Paste │ │ (renders nodes = events, │ │
│ │ history │ │ edges = causal links) │ │
│ │ here" │ │ │ │
│ └──────────────┘ │ User drags a node ───────┐ │ │
│ └─────────────────────────────┼─────────┘ │
└──────────────────────────────────────────────────────────┼──────────────┘
│ POST /extract │ WS: node_moved
▼ ▼
┌─────────────────────────────────────────────────────────────────────┐
│ FASTAPI BACKEND │
│ │
│ ┌────────────────────┐ ┌───────────────────────────────────┐ │
│ │ EXTRACTION ENGINE │ │ SPATIAL INTELLIGENCE ENGINE │ │
│ │ (Chapter 2) │ │ (Chapter 3 — the heart of it) │ │
│ │ │ │ │ │
│ │ 1. Split into │ │ 1. Build a graph: nodes=events, │ │
│ │ sentences │ │ edges=causal relations │ │
│ │ 2. Run NER │────▶│ 2. Init x,y for every node as a │ │
│ │ (BERT) to find │ │ TRAINABLE TENSOR │ │
│ │ events, dates, │ │ (requires_grad=True) │ │
│ │ people, places │ │ 3. Define "Map Clarity Loss": │ │
│ │ 3. Run Relation │ │ - causally linked nodes should │ │
│ │ Extraction to │ │ be CLOSE │ │
│ │ find "A causes B" │ │ - unrelated nodes should be │ │
│ │ 4. Output: list of │ │ FAR (repulsion) │ │
│ │ (event, date, │ │ - nothing should overlap │ │
│ │ relation, event) │ │ 4. Run backprop on the │ │
│ │ triples │ │ COORDINATES (not the model's │ │
│ │ │ │ weights!) using a hand-written │ │
│ │ │ │ Adam optimizer │ │
│ │ │ │ 5. When user drags a node, PIN │ │
│ │ │ │ that node's coords, re-run │ │
│ │ │ │ optimization on everyone else │ │
│ └────────────────────────┘ └───────────────────────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌────────────────────────────────────────────────────────────────┐ │
│ │ PostgreSQL + pgvector (stores triples + embeddings) │ │
│ └────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────┘
Enter fullscreen mode Exit fullscreen mode
Data Flow: What Happens When a User Clicks "Generate"
Walk through it end to end, because this sequence is the skeleton every later chapter hangs off of:
- User pastes text (say, three paragraphs on the causes of WWI) and clicks Generate.
-
Frontend POSTs the raw text to
/api/extract. - Backend NER model (a fine-tuned BERT) tags every token: is it part of an EVENT, a DATE, a PERSON, a PLACE? Output: a list of entity spans.
-
Backend Relation Extraction model looks at pairs of entities in the same or nearby sentences and predicts a relation label:
CAUSES,ENABLED_BY,PRECEDES,PART_OF, orNONE. - The result is a set of triples:
(Assassination of Franz Ferdinand, CAUSES, Austria-Hungary ultimatum to Serbia). These get written to Postgres, and each event also gets a text embedding stored via pgvector (this lets us later cluster thematically similar events even without an explicit extracted relation). - The triples define a graph: events are nodes, relations are edges.
-
The Spatial Intelligence Engine initializes every node at a random (x, y) position — as a PyTorch tensor with
requires_grad=True. This is the part that makes ChronoWeave unusual: the coordinates themselves are the parameters being learned, not neural network weights. - We define a custom loss function — the Map Clarity Loss — that penalizes bad layouts: causally-linked nodes that are far apart, unrelated nodes that overlap, edges that cross each other unnecessarily.
- We run gradient descent (a hand-written Adam optimizer) for a few hundred steps. Each step: compute the loss, call
.backward(), get gradients with respect to the x,y coordinates, nudge the coordinates a little in the direction that reduces loss. Repeat. - After convergence, the backend streams the final
(node_id, x, y)positions back to the frontend over the initial REST response (or WebSocket, for the live re-layout case). - React + D3.js renders nodes at those coordinates, with edges drawn between causally-linked events, styled by relation type.
-
The magic moment: the user drags a node to a new spot because they disagree with the layout, or just want to explore. That node's position gets pinned (
requires_grad=Falsefor it, fixed at the drag target). The backend re-runs the optimizer on every other node, live, streaming intermediate frames over a WebSocket, so the user watches the rest of the graph flow and resettle around their edit — like iron filings reorganizing around a magnet you just moved.
That last step is the entire reason this project exists. A physics engine (force-directed graph layout, e.g. D3's built-in forceSimulation) can also do node repulsion and edge attraction — but it can't easily express arbitrary, learned, semantically-weighted objectives (e.g., "cluster by decade AND by causal chain AND penalize edge crossings, with weights that adapt based on graph density"). Gradient descent on a custom loss can. That's the pitch. We'll come back to defending it rigorously in Chapter 3.
Technology Stack: Why Each Piece Was Chosen
| Layer | Choice | Why |
|---|---|---|
| ML core | PyTorch (from raw tensors, then nn.Module later) |
You need autograd exposed at the tensor level to do coordinate optimization; PyTorch's dynamic graph makes this natural |
| NER / Relation Extraction | Fine-tuned BERT-base, later maybe REBEL for joint extraction | BERT is small enough to fine-tune on a single GPU and well documented; REBEL is a strong upgrade path for joint entity+relation extraction |
| Backend | FastAPI + WebSockets | Async-native, typed, and WebSocket support is first-class for the live re-layout streaming |
| Database | PostgreSQL + pgvector | Relational structure for triples, vector column for semantic similarity search on event embeddings |
| Frontend | React + D3.js | D3 gives you full control over force/position rendering; React manages component state and drag events |
| Deployment | Docker Compose → cloud (Render/Fly.io/AWS) | Reproducibility first, then scale |
The Journey Ahead
Four phases, each roughly a month, building strictly bottom-up:
- Phase 1 (Weeks 1–4): You build the math — autograd, manual backprop, hand-rolled optimizers, a toy Transformer. No frameworks doing the work for you yet.
- Phase 2 (Weeks 5–8): You build the reader — BERT-based NER and relation extraction, a real data pipeline, a real database.
- Phase 3 (Weeks 9–12): You build the cartographer — the spatial intelligence engine. This is the conceptual core of the whole project and gets the most depth in this guide.
- Phase 4 (Weeks 13–16): You build the body — the full-stack app, the live drag-and-reoptimize loop, and you ship it.
🧭 The Mentor Says: Don't let the ambition of the final product distract you from the ugliness of the early steps. In week 2 you'll be manually computing gradients for a two-layer network with a pencil-and-paper feel to it, and it will seem impossibly far from "beautiful interactive graph app." That gap is supposed to be there. Every person who deeply understands PyTorch went through exactly this tunnel. You don't skip it by using nn.Module early — you skip understanding by doing that.
CHAPTER 1: The Foundation — Building Your Neural Engine
(Weeks 1–4 · Prerequisite: comfort with Python, basic linear algebra — matrix multiply, dot products — and derivatives from a first calculus course)
Week 1: Environment and the Shape of a Tensor
The Concept
A tensor is just a multi-dimensional array with two superpowers bolted on: it knows how to run on a GPU, and it can remember the sequence of operations that produced it, so it can later compute how a tiny nudge to its inputs would change its output. That second superpower is autograd, and it's the single idea Phase 1 exists to demystify.
Think of a tensor with requires_grad=True as a spreadsheet cell that doesn't just hold a number — it holds a number and a formula referencing other cells. Change an input cell, and every downstream cell knows exactly how much it would change, without you re-deriving the formula by hand. That "how much would it change" is the gradient.
The Code Challenge
Set up your environment first:
poetry init
poetry add torch numpy matplotlib jupyter
poetry shell
Enter fullscreen mode Exit fullscreen mode
Then in a notebook, don't build anything yet — just observe autograd:
import torch
# TODO: create a tensor x = 3.0 with requires_grad=True
x = ...
# TODO: create y = x**2 + 2*x + 1
y = ...
# TODO: call y.backward() and print x.grad
# Predict on paper what dy/dx should be at x=3 BEFORE you run this
...
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
x.grad should equal 2*x + 2 = 8.0 — exactly the calculus-class derivative, computed automatically. The click isn't "wow it did calculus," it's realizing PyTorch didn't look up a symbolic formula for x**2 + 2*x + 1 — it recorded the sequence of primitive operations (pow, mul, add) as you executed them, and walked that sequence backward, applying the chain rule at each step. This recorded sequence is called the computation graph, and it's rebuilt fresh every time you run forward — which is why PyTorch is called "define-by-run."
The Extension
Every gradient ChronoWeave ever computes — whether it's a loss with respect to a BERT weight, or a loss with respect to a node's (x, y) position on the canvas — goes through this exact mechanism. There is no different code path for "coordinates" versus "weights." This is the whole trick behind Phase 3, twelve weeks from now.
Resources
- PyTorch autograd tutorial: https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html
🕳️ The Rabbit Hole: Read about torch.autograd.grad() vs .backward() — the former lets you compute gradients without accumulating them into .grad, important later when you want gradients with respect to coordinates specifically, while leaving model weights untouched.
✅ Checkpoint 1.1: Explain, without looking anything up, why calling .backward() twice on the same graph (without retain_graph=True) throws an error. (Answer: the graph is freed after the first backward pass to save memory.)
Week 2: Manual Forward and Backward — No nn.Module
The Concept
nn.Module is a convenience wrapper. Underneath, a "layer" is nothing more than: take an input, multiply by a weight matrix, add a bias, pass through a nonlinearity. A "network" is several of those chained. "Training" is: compute a loss, get its gradient with respect to every weight, nudge each weight opposite to its gradient. Building this by hand once is the single highest-leverage exercise in this project.
The Code Challenge
Fit y = sin(x) on x in [-π, π] with manual parameters:
import torch
torch.manual_seed(0)
# Architecture: 1 -> 32 -> 1, tanh activation
W1 = ... # TODO: (1,32), requires_grad=True, small random init
b1 = ... # TODO: (32,)
W2 = ... # TODO: (32,1)
b2 = ... # TODO: (1,)
def forward(x):
# TODO: z1 = x @ W1 + b1 ; a1 = tanh(z1) ; z2 = a1 @ W2 + b2
...
return z2
def mse_loss(pred, target):
# TODO: mean squared error, raw tensor ops
...
lr = 0.05
for step in range(2000):
x = torch.linspace(-3.14, 3.14, 200).unsqueeze(1)
y_true = torch.sin(x)
pred = forward(x)
loss = mse_loss(pred, y_true)
# TODO: zero old grads, loss.backward(), manually update all 4 params
# (wrap update in `with torch.no_grad():`)
...
if step % 200 == 0:
print(step, loss.item())
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
Loss will not decrease if you forget to zero gradients — PyTorch accumulates them into .grad by default. Once burned by this, you'll never forget zero_grad() again — you'll understand why, not just cargo-cult it. Then plot forward(x) against sin(x): a sine wave emerges from 4 matrices you initialized as noise.
The Extension
This loop — forward, loss, zero grad, backward, manual update — is structurally identical to the loop you write in Phase 3 for optimizing node coordinates. Only what the "parameters" represent changes.
😤 The Struggle: You'll likely hit RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn, usually from overwriting a leaf tensor without torch.no_grad(). Print .requires_grad / .is_leaf on every parameter right before the failing line.
Resources
- PyTorch neural network basics: https://docs.pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html (concepts only this week)
- 3Blue1Brown, "Backpropagation calculus"
Week 3: Hand-Rolled SGD, Momentum, RMSprop, and Adam
The Concept
An optimizer answers: given a gradient, how exactly should I change the parameter? Plain SGD (subtract lr*grad) works but is slow and oscillates on ravine-shaped losses.
- Momentum — running average of past gradients, damping oscillation.
- RMSprop — running average of squared gradients per parameter; divides the step by its square root, shrinking effective learning rate for consistently-large-gradient parameters.
- Adam — momentum + RMSprop, plus bias-correction for early steps.
The Code Challenge
class ManualSGD:
def __init__(self, params, lr=0.01, momentum=0.0):
self.params = list(params)
self.lr = lr
self.momentum = momentum
self.velocities = ... # TODO: zeros_like buffer per param
def step(self):
with torch.no_grad():
for p, v in zip(self.params, self.velocities):
if p.grad is None: continue
# TODO: v = momentum*v - lr*p.grad ; p += v
...
def zero_grad(self):
for p in self.params:
if p.grad is not None: p.grad.zero_()
class ManualAdam:
def __init__(self, params, lr=0.001, betas=(0.9, 0.999), eps=1e-8):
self.params = list(params)
self.lr, self.b1, self.b2, self.eps, self.t = lr, *betas, eps, 0
self.m = ... # TODO: zeros_like per param (first moment)
self.v = ... # TODO: zeros_like per param (second moment)
def step(self):
self.t += 1
with torch.no_grad():
for i, p in enumerate(self.params):
if p.grad is None: continue
g = p.grad
# TODO: m[i] = b1*m[i] + (1-b1)*g
# TODO: v[i] = b2*v[i] + (1-b2)*g**2
# TODO: m_hat = m[i]/(1-b1**t) ; v_hat = v[i]/(1-b2**t)
# TODO: p -= lr * m_hat / (sqrt(v_hat) + eps)
...
def zero_grad(self):
for p in self.params:
if p.grad is not None: p.grad.zero_()
Enter fullscreen mode Exit fullscreen mode
Re-run the Week 2 sine-fit with ManualAdam, compare convergence speed to vanilla SGD.
The Aha Moment
Adam converges in hundreds of steps where SGD needed thousands, and is far less sensitive to learning rate choice. This robustness is why you'll reach for it again in Phase 3, where the loss surface (Map Clarity Loss over 2D positions) is even messier.
The Extension
Your Phase 3 CoordinateAdam class will be a near copy of this one — same math, different thing being optimized.
💡 The Innovation (early preview): "I wrote my own Adam optimizer for coordinate updates because I needed to freeze individual coordinates mid-optimization when a user grabs a node — the standard torch.optim API doesn't expose that cleanly." Keep this for Chapter 3.
Resources
- Adam paper (Kingma & Ba, 2014): https://arxiv.org/abs/1412.6980
- Sebastian Ruder, "An overview of gradient descent optimization algorithms": https://www.ruder.io/optimizing-gradient-descent/
✅ Checkpoint 1.2: Plot loss curves for SGD, SGD+momentum, RMSprop, Adam — same problem, same init, same step count. If Adam isn't fastest, check bias-correction first.
Week 4: A Transformer From Scratch (Tiny, But Real)
The Concept
A Transformer's core operation: for every token, compute a weighted average of every other token's representation, where weights ("attention") are learned and reflect relevance. Everything else — multi-head, positional encoding, layer norm, feedforward — is scaffolding.
Analogy: a room full of people (tokens) forming an opinion on a topic — you weight each person's input by relevance. Query = what you're looking for; Key = what each token offers; Value = what it contributes if attended to.
The Code Challenge
import torch, math
import torch.nn.functional as F
def scaled_dot_product_attention(Q, K, V, mask=None):
d_k = Q.shape[-1]
scores = ... # TODO: Q @ K.T / sqrt(d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, float('-inf'))
weights = ... # TODO: softmax(scores, dim=-1)
output = ... # TODO: weights @ V
return output, weights
# Sanity check: verify weights.sum(dim=-1) is all 1.0
Enter fullscreen mode Exit fullscreen mode
class TinyMultiHeadAttention(torch.nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
assert d_model % n_heads == 0
self.d_k, self.n_heads = d_model // n_heads, n_heads
... # TODO: four Linear layers W_q, W_k, W_v, W_o
def forward(self, x, mask=None):
... # TODO: project, reshape to heads, attend per head, concat, W_o
class TinyEncoderLayer(torch.nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.attn = TinyMultiHeadAttention(d_model, n_heads)
self.norm1 = torch.nn.LayerNorm(d_model)
self.ff = torch.nn.Sequential(
torch.nn.Linear(d_model, d_ff), torch.nn.ReLU(),
torch.nn.Linear(d_ff, d_model))
self.norm2 = torch.nn.LayerNorm(d_model)
def forward(self, x, mask=None):
... # TODO: x = norm1(x + attn(x)) ; x = norm2(x + ff(x))
Enter fullscreen mode Exit fullscreen mode
Train on next-character prediction over a small corpus (a few historical-text paragraphs fits the theme).
The Aha Moment
Visualize attention weights for one sentence as a heatmap — rows should light up on semantically relevant tokens (e.g. a pronoun attending to the noun it refers to). Attention stops being a diagram and becomes something you watch your model do.
The Extension
In Chapter 2 you'll swap this toy encoder for a pretrained BERT, understanding exactly what happens inside every layer because you built the smallest version yourself.
Resources
- "The Annotated Transformer" (Harvard NLP): https://nlp.seas.harvard.edu/annotated-transformer/
- "Attention Is All You Need": https://arxiv.org/abs/1706.03762
- Jay Alammar, "The Illustrated Transformer": https://jalammar.github.io/illustrated-transformer/
🕳️ The Rabbit Hole: Attention has no inherent sense of token order. Read the sinusoidal positional encoding and ask why sine/cosine rather than a learned embedding.
✅ Checkpoint 1.3 — End of Phase 1: From memory, sketch a Transformer encoder layer's forward pass, and explain what breaks if you remove residual connections, layer norm, or the sqrt(d_k) scaling.
Phase 1 Milestone
By end of Week 4, from raw tensor operations, you have: (1) intuitive autograd understanding, (2) a hand-built 2-layer network with manual gradient updates, (3) three hand-built optimizers, (4) a working tiny Transformer with visualized attention. Your Week 3 optimizers become the Phase 3 coordinate optimizer almost verbatim; your Week 4 Transformer understanding is what lets you debug BERT rather than treat it as a black box.
CHAPTER 2: The Understanding — Teaching Your Engine to Read History
(Weeks 5–8 · Prerequisite: Phase 1 complete, comfort reading model architecture diagrams)
Week 5: Fine-Tuning BERT for Named Entity Recognition
The Concept
Your tiny Transformer from Week 4 learned character prediction from scratch on almost no data. BERT is the same architecture, scaled up (12 layers, 12 heads, 768-dim), pretrained on billions of words. Fine-tuning means: keep the pretrained weights as a strong starting point, attach a small task-specific head on top (here, a linear layer mapping each token's final hidden state to a label like B-EVENT, I-EVENT, B-DATE, O), and train the whole thing (or just the head) on your labeled data for a few epochs.
The reason this works with so little labeled data compared to training from scratch: BERT already learned general-purpose language structure from pretraining (grammar, word relationships, some world knowledge). Fine-tuning just teaches it to route that existing knowledge toward your specific labeling scheme.
The Code Challenge
from transformers import BertTokenizerFast, BertForTokenClassification
import torch
# TODO: load "bert-base-cased" tokenizer and BertForTokenClassification
# with num_labels = len(your_label_list)
# e.g. label_list = ["O","B-EVENT","I-EVENT","B-DATE","I-DATE","B-PERSON","I-PERSON"]
tokenizer = ...
model = ...
def tokenize_and_align_labels(text, word_labels):
"""
text: list of words, e.g. ["The", "assassination", "of", "Franz", "Ferdinand"]
word_labels: list of label ids, one per word
Returns tokenized input + labels aligned to WordPiece subtokens
(subtokens after the first get label -100 so they're ignored in the loss)
"""
# TODO: tokenizer(text, is_split_into_words=True, ...) then use
# .word_ids() to map each subtoken back to its source word
...
# TODO: set up a small labeled dataset (start with ~50-100 hand-labeled
# sentences from historical text — yes, you label it yourself first)
# TODO: standard fine-tuning loop: forward, CrossEntropyLoss (ignore_index=-100),
# backward, optimizer.step() -- use torch.optim.AdamW this time, not your manual one,
# since you've already proven you understand what it's doing
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
Run inference on a sentence the model never saw during fine-tuning and watch it correctly tag "the outbreak of war" as an EVENT span and "1914" as a DATE span, purely from ~100 examples. That's the pretraining transfer working — this would be essentially impossible to get right training from random initialization on 100 sentences.
The Extension
This NER output — spans tagged as EVENT, DATE, PERSON, PLACE — is exactly what Week 6's relation extraction will pair up into causal triples.
Resources
- Hugging Face token classification guide: https://huggingface.co/docs/transformers/tasks/token_classification
- BERT paper (Devlin et al., 2018): https://arxiv.org/abs/1810.04805
😤 The Struggle: Label alignment between words and WordPiece subtokens is the single most common source of silent bugs in NER fine-tuning — a misaligned label doesn't crash, it just quietly trains the model on garbage. Always spot-check tokenizer.convert_ids_to_tokens() against your aligned label array for a handful of examples before you trust any training run.
Week 6: Relation Extraction — Finding "A Causes B"
The Concept
NER tells you what the entities are. Relation extraction tells you how they relate. The simplest approach: for every pair of entities that co-occur in the same sentence (or a short window of sentences), feed their contextualized representations (plus the sentence itself) into a classifier that predicts one of a fixed set of relation labels: CAUSES, ENABLED_BY, PRECEDES, PART_OF, or NONE.
A more powerful approach — REBEL — does entity and relation extraction jointly, generating the full set of triples as a structured text sequence in one pass, rather than requiring you to first extract entities, then classify every pair. It's a strong upgrade path once your simpler pairwise classifier is working and you understand why the joint approach is more efficient (it doesn't blow up combinatorially with the number of entities in a sentence).
The Code Challenge
Start with the pairwise classifier — it's the better learning exercise, even though REBEL is more production-capable:
# For each entity pair (e1, e2) in the same sentence:
# 1. Mark their spans in the input with special tokens, e.g.
# "[E1] The assassination [/E1] of Franz Ferdinand triggered
# [E2] Austria-Hungary's ultimatum [/E2] to Serbia."
# 2. Run through BERT
# 3. TODO: take the hidden states at the [E1] and [E2] marker positions,
# concatenate them, pass through a small classifier head
# -> softmax over {CAUSES, ENABLED_BY, PRECEDES, PART_OF, NONE}
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
The output for that Franz Ferdinand sentence should be CAUSES with high confidence. Then feed it a sentence with two entities that are merely mentioned near each other with no causal link ("The war began in 1914. Assassinations were common in the region.") and watch it correctly predict NONE. That contrast — the model discriminating relevance, not just proximity — is the whole point of relation extraction over naive "entities near each other are related" heuristics.
The Extension
Every (entity_1, relation, entity_2) triple this produces becomes an edge in the causal graph that Phase 3's spatial engine will lay out.
Resources
- REBEL paper (Huguet Cabot & Navigli, 2021): https://aclanthology.org/2021.findings-emnlp.204/
- REBEL on Hugging Face: https://huggingface.co/Babelscape/rebel-large
🕳️ The Rabbit Hole: Read about "distant supervision" for relation extraction — a technique for auto-generating noisy training labels by aligning a knowledge base (like Wikidata) against text, instead of hand-labeling everything. Useful if your ~100 hand-labeled examples aren't enough once you scale up.
Week 7: The Data Pipeline — Text → Tokens → Entities → Relations → Storage
The Concept
A pipeline is only as trustworthy as its weakest stage boundary. This week isn't about new ML — it's about wiring Weeks 5 and 6 together into something that reliably takes raw pasted text and emits a clean, deduplicated, database-ready set of triples, with sane error handling for the inevitable garbage input (empty text, non-English text, text with no extractable events).
The Code Challenge
class ExtractionPipeline:
def __init__(self, ner_model, relation_model, tokenizer):
...
def extract(self, raw_text: str) -> list[dict]:
"""
Returns: [{"subject": "...", "relation": "CAUSES", "object": "...",
"subject_date": "1914-06-28" or None, ...}, ...]
"""
# TODO Step 1: sentence-split raw_text (spaCy or nltk sentence tokenizer)
# TODO Step 2: run NER on each sentence -> entity spans
# TODO Step 3: for entity pairs within a sentence AND across
# adjacent sentences (causal claims often span 2 sentences),
# run the relation classifier
# TODO Step 4: filter relation predictions below a confidence
# threshold (start at 0.6, tune empirically)
# TODO Step 5: deduplicate triples (same subject/object/relation
# appearing from overlapping sentence windows)
# TODO Step 6: return structured triples
...
Enter fullscreen mode Exit fullscreen mode
Set up PostgreSQL with pgvector:
CREATE EXTENSION IF NOT EXISTS vector;
CREATE TABLE events (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
text TEXT NOT NULL,
date_text TEXT,
embedding VECTOR(768), -- BERT's hidden size
created_at TIMESTAMPTZ DEFAULT now()
);
CREATE TABLE relations (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
source_event_id UUID REFERENCES events(id),
target_event_id UUID REFERENCES events(id),
relation_type TEXT NOT NULL,
confidence FLOAT
);
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
Paste in three unrelated paragraphs of history (say, one on WWI causes, one on the French Revolution, one on the fall of Rome) and watch the pipeline correctly keep the triples from each topic separate, with no spurious cross-topic relations — because relation extraction is confidence-thresholded and entity embeddings from unrelated topics don't get spuriously matched.
The Extension
This pipeline is the entire backend of the /api/extract endpoint from Chapter 0's data flow diagram. Everything downstream (Phase 3, Phase 4) consumes its output.
Resources
- pgvector docs: https://github.com/pgvector/pgvector
- SQLAlchemy + pgvector integration: https://github.com/pgvector/pgvector-python
✅ Checkpoint 2.1: Run your pipeline on a genuinely messy input (a Wikipedia paragraph with footnote markers, inconsistent date formats, nested clauses) and verify it degrades gracefully — no crash, just fewer/lower-confidence triples — rather than throwing an unhandled exception.
Week 8: Evaluation and Hardening
The Concept
An extraction pipeline that "looks right" on your three favorite test paragraphs is not the same as one that's reliable. This week is about building a small held-out evaluation set (hand-labeled, separate from training data) and computing real metrics: precision, recall, and F1 for both entity spans and relation classification.
The Code Challenge
def evaluate_ner(model, eval_sentences, eval_labels):
# TODO: run inference, compute span-level (not token-level!) precision/recall/F1
# Span-level means a predicted "B-EVENT I-EVENT" span only counts as
# correct if it matches the gold span's exact boundaries, not just
# individual token labels — this is a stricter and more meaningful metric.
...
def evaluate_relations(model, eval_pairs, eval_labels):
# TODO: per-class precision/recall/F1, plus a confusion matrix
# (CAUSES vs ENABLED_BY vs PRECEDES are semantically close and
# commonly confused — expect this)
...
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
Your confusion matrix will likely show CAUSES and ENABLED_BY bleeding into each other — this is expected and informative, not a bug. It tells you those two categories are genuinely hard to distinguish from surface text alone (a human annotator would disagree with themselves on some of these too), which should inform how confidently the frontend displays that distinction later (e.g., maybe you merge them into one edge style with a subtler visual difference rather than two starkly different arrow types).
The Extension
Phase 2 milestone: a real, measured, imperfect-but-quantified extraction system, feeding real triples into a real database.
Resources
- "Named Entity Recognition" evaluation conventions (seqeval library): https://github.com/chakki-works/seqeval
Phase 2 Milestone
You now have a fine-tuned BERT NER model, a relation extraction classifier, a hardened pipeline connecting them, a Postgres+pgvector store, and honest evaluation metrics on held-out data. This is a legitimate, demoable NLP system on its own — worth pausing to appreciate before diving into Phase 3, which is where the project becomes genuinely novel.
CHAPTER 3: The Intelligence — Making Your Engine Think Spatially
(Weeks 9–12 · Prerequisite: Phase 1 and 2 complete. This is the conceptual core of ChronoWeave — take this chapter slowly.)
Why This Chapter Gets the Most Depth
Everything before this point — the Transformer, the NER model, the relation extractor — is, architecturally, "standard" deep learning applied to a specific domain. Impressive to build from scratch, but conceptually well-trodden. This chapter is where ChronoWeave does something genuinely uncommon: using gradient descent as a general-purpose layout algorithm, with coordinates as first-class trainable parameters. Take this slowly. It's the part of the project you'll actually be excited to explain in an interview.
Week 9: Coordinates as Parameters
The Concept
Every optimization problem in ML has the same shape: parameters, a loss function measuring how bad the current parameters are, and an optimizer that nudges parameters to reduce that loss. So far, "parameters" has meant neural network weights. Nothing in that shape requires the parameters to be weights. A parameter is just a tensor with requires_grad=True that appears somewhere in a differentiable computation whose output is your loss.
So: what if the parameters are the (x, y) positions of graph nodes on a canvas, and the loss is a hand-designed function that scores how "good" a layout is?
This reframes graph layout — traditionally solved with physics simulations (force-directed layout, spring-embedder algorithms) — as an optimization problem solvable with the exact same machinery you built in Phase 1.
The Code Challenge
Set up the skeleton, no loss function yet — just prove coordinates can be optimized at all:
import torch
num_nodes = 10
# TODO: initialize positions as a (num_nodes, 2) tensor, random in
# some reasonable range (e.g. -5 to 5), requires_grad=True
positions = ...
# Toy goal: pull every node toward the origin (0,0) — trivial loss,
# just to prove the plumbing works before you write anything smarter
def toy_loss(positions):
# TODO: sum of squared distances from origin
return (positions ** 2).sum()
optimizer = torch.optim.Adam([positions], lr=0.1)
for step in range(100):
optimizer.zero_grad()
loss = toy_loss(positions)
loss.backward()
optimizer.step()
if step % 20 == 0:
print(step, loss.item())
# TODO: plot positions before and after — every node should have
# collapsed toward (0,0)
Enter fullscreen mode Exit fullscreen mode
The Aha Moment
Watching ten random points converge toward the origin using torch.optim.Adam — the exact same optimizer class you'd use to train a neural network — is the moment this project's central idea stops being abstract. You just used backpropagation to solve a geometry problem. No physics, no forces, no simulation of springs — just calculus.
The Extension
toy_loss is a stand-in. Next week you replace it with the real Map Clarity Loss — but the training loop around it doesn't change at all.
🧭 The Mentor Says: Resist the urge to jump straight to the full Map Clarity Loss this week. Build up loss terms one at a time, verify each one does what you expect in isolation (attraction only, then repulsion only, then combined), and only then combine everything. Debugging a five-term loss function that's never converged correctly at any point is miserable. Debugging a five-term loss function where you've verified each term separately is straightforward.
Week 10: The Map Clarity Loss
The Concept
A good layout satisfies several competing objectives simultaneously:
- Attraction: causally-linked events should be close together (short edges are more readable)
- Repulsion: all events should push apart from each other generally, so the graph doesn't collapse into a single point (this is what "attraction" alone would do — see Week 9's toy example)
- Non-overlap: no two event boxes should visually overlap
- Temporal ordering (optional but powerful): since these are historical events, you may want an additional soft constraint that events with earlier dates trend toward one side of the canvas — turning the layout into something between a pure causal
0 Comments
Log in to join the conversation.No comments yet. Be the first to share your thoughts.