No session running.
AI Safety Camp 12 · Project proposal · Virtual, Jan–Apr 2027
Delta Drills is an adaptive tutor for the prerequisites of AI safety research engineering. It maps what a learner already knows onto a knowledge graph of the ARENA curriculum, teaches only what is missing, and keeps it from decaying.
82 concepts on a prerequisite graph, 1,585 executable drills, a graph-wide placement test, spaced review, and remediation that walks down the graph.
Do ARENA organizers discover prerequisite gaps only after a cohort starts? When learners stall mid-course, is the cause a missing prerequisite?
Whether a shorter ramp translates into more safety research, and whether organizers will change their workflow to use it. The AISC project is designed to find out.
01 · The bottleneck
ARENA is the standard on-ramp from "motivated and numerate" to "can implement a transformer and run interpretability experiments." Its exercises assume fluency in PyTorch tensor manipulation, einops, einsum, linear algebra and probability. Cohorts arrive with very different amounts of that fluency.
Our working hypothesis is that two things go wrong at the start of the ramp. Readiness is guessed, not measured: organizers learn who was under-prepared only after the course begins. And when a learner stalls, the real cause is usually upstream — an exercise on attention fails because a broadcasting rule never stuck — so the help they get (a TA, an LLM, re-reading the notebook) treats the symptom.
These are hypotheses, not findings. The next step is interviewing ARENA learners, facilitators and organizers about what actually happened in their last cohort; what we learn goes in § Evidence rather than asking anyone whether they would like the tool.
02 · Theory of change
Why this is safety work and not ed-tech, in three legs. None of them is shown by data yet; each names what would test it.
You can't bootstrap trust in an AI that may be manipulating you. Control helps, but the last line is a human with enough technical understanding to judge what the AI says and catch it trying to deceive. Human review is expensive; a reviewer who actually understands the model's code and claims reviews better and faster. Upskilling raises the quality of oversight and lowers its cost.
AI research effort grows ~25× a year; human research effort ~4%. We can't steepen the human curve. We can shorten the time it takes to become a useful safety researcher, which grows the pool and moves researcher-years into the window where they matter. Experts benefit too, as new sub-fields and tools appear.
Dual use is real: in capabilities hands the same tool speeds up capabilities. So Delta Drills is non-profit by design (Math Academy is for-profit, ~$50/month) and deliberately passes on capabilities money, and it is distributed only through safety networks: GEAR, AISC, local AI-safety groups, with a curriculum scoped to ARENA.
Why the green line matters: humans who understand the system → align the trusted model → which oversees the untrusted model doing the work. Understanding is what lets humans hold it accountable and correct it.
03 · How it works
The architecture follows Math Academy's published design — a knowledge graph, a student model, a diagnostic, and spaced repetition that understands how skills contain one another. We rebuilt it for a different domain: instead of multiple-choice mathematics, learners write PyTorch that is executed and graded, and the graph is the prerequisite structure of the ARENA curriculum. Every figure below runs the same update rules as the production backend.
Read ARENA in order. Drills patch the gaps. The ARENA notebooks themselves are built into Delta Drills and read section by section, in order; ARENA stays the course. Under every exercise sits one button, e.g. Practice make_rays_1d: it starts a short timed block of drills on the concepts that exercise uses. A miss inside the block pulls drills on that concept's direct prerequisites to the front of the queue. When the block ends, the learner goes back to the notebook.
Each node is a concept a learner can be tested on; each arrow says "you cannot do this without that." Grey nodes are prerequisites, and nodes are coloured by the lesson they belong to. Click any concept to light up everything it depends on and everything that depends on it.
A learner's knowledge frontier is the boundary between what they know and what they are ready to learn next. Testing all 82 concepts one by one would take a day. Instead, every answer moves belief across the graph: passing a concept is evidence you know its prerequisites; failing one is evidence against what depends on it. The test always asks about the concept whose answer is expected to remove the most uncertainty across the whole graph. Play the learner below.
Memory decays. Reviewing just as recall starts to slip resets the curve and makes the next decay slower, so each well-timed review buys more time than the last. Drag the review markers, or let the scheduler place them at the retention target. Learners and topics differ in speed: a concept that comes easily needs fewer reviews; a hard one needs more, sooner.
Built vs. planned. Delta Drills today decays every belief with a fixed 14-day half-life: a faded concept rises in the queue and resurfaces, but a review does not lengthen the next interval. Expanding, per-learner intervals are what Math Academy has and what the AISC team would fit from real logs. Switch between the two below.
Flashcard schedulers treat every fact as independent. Skills are not: writing a reduction along an axis exercises broadcasting and aggregation. When the graph says a concept encompasses a simpler one, a correct answer on the advanced concept credits the simpler one with a fractional repetition. Due reviews can then be knocked out by new learning — the fewest pushes to topple the most dominoes.
The usual response to a miss is an easier problem on the same concept. That keeps the learner on the symptom. After two misses since their last unaided success, Delta Drills redirects to the weakest prerequisite that still has fresh drills, serves a dose of three, then returns. Interleaving is enforced: no concept more than three times in a row while other frontier work exists.
04 · What is new here
We credit Math Academy for the architecture and borrow its best explanations. What we built on top is specific to research engineering, and several pieces are original.
| Math Academy | Delta Drills | |
|---|---|---|
| Domain | School and university mathematics | The prerequisites of ARENA: PyTorch, einops, einsum, linear algebra, probability |
| Answers | Numeric / multiple choice | Code, executed in a sandbox and checked against test cases; near-miss feedback for wrong dtypes, shapes and calls |
| Student model | Knowledge profile of spaced repetitions | Per-concept BKT with implicit credit plus an additive-logistic model (generalising IRT and Elo/Glicko) that gives P(correct) with a credible interval |
| Placement | Question-count diagnostic | Time-budgeted (1 h / 3 h / 6 h), value-weighted by how much of the course each concept unlocks, probes with ARENA's own exercises first, and reports prerequisite edges your answers contradict |
| Readiness | Mastery gates on prerequisites | Before each drill, asks whether the lower bound of P(correct) clears the bar — and whether re-reading the lesson would lift it. If so, the lesson comes back first |
| Relationship to a curriculum | Is the curriculum | An adaptive layer around an existing one: ARENA stays the course; we get people ready for it and unstuck inside it |
| Access | Paid subscription | Free for the safety community |
05 · Project plan & fit for AISC
The software is mostly done; the remaining work is content. Deliverable: the whole ARENA curriculum mapped, as a knowledge graph plus verified drills, by the end of April.
The team is the first cohort. Every teammate works through ARENA inside Delta Drills as a learner and reports where the content or the graph is wrong; the team revises both as it goes. Content is written just in time, one chapter ahead, because ARENA's material already exists and each later chapter gets authored with a stronger model. Teammates aren't neutral users, so an outside group still matters for evidence.
Chapter by chapter, just in time: concepts, prerequisite edges, encompassing weights, and drills that pass the existing content gates. Seth is the main author; an ML-strong teammate reviews and cleans what he generates.
Seth leads. Fit BKT and logistic-engine parameters to the team's own attempt logs; publish calibration (Brier score, reliability curves); replace the fixed half-life with fitted memory stability; log every lesson read.
Teammates' learning as the first data, plus an outside group (GEAR cohort, local AI-safety group). A quantitative A/B comparison of learning outcomes: ARENA with Delta Drills vs. the usual way (notebooks in Colab), with the same checkpoint tests given to both arms during the course and at the end; observe, don't coach.
06 · What counts as evidence
We are deliberately not collecting "would you use this?" answers. Interviews ask about the last cohort: how readiness was judged, the last participant who fell behind, and what the facilitator actually did about it. The product is shown only after the problem is understood, and conversations end with a request for a concrete next step.
The first learner is the project lead. Below is every graded answer, placement probe and lesson read from September on the ARENA prerequisites, pulled from the production log and scored three ways. It is one person, so it proves nothing about anyone else; what it does show is where the model is wrong, which is where Stream 2 starts.
≈48 h studied in the app since Jul 21.
Live app; 14 lessons, 82 concepts, 1,585 drills; engine, placement and remediation; ARENA notebooks in-app.
EAGx Berkeley: 7–8 people tried it when it wasn't ready (weak content, confusing UX). Since the rebuild, one outside learner, and he chose to do ARENA in Delta Drills instead of Google Colab. That is a commitment, not a compliment, but he is a friend: n = 1, biased. The only sustained user is the author (above).
None formal yet. Next: the local AI-safety group, then the GEAR online ARENA cohort (Oct–Dec, applying now), where the author works through ARENA inside Delta Drills alongside other learners. Questions are about their last stuck moment, not about the product.
Completion of the diagnostic; whether it separates learners meaningfully; whether recommendations change what they study; whether organizers use the result.
Triggered at meaningful moments; completed; learner returns to the ARENA exercise and progresses; less facilitator time; voluntary re-use.
Return rate without reminders; where learners stop; calibration of predicted vs. actual correctness.
07 · Uncertainties & downside risk
Beyond saying they're interested. Today: 7–8 EAGx testers when it wasn't ready, one outside learner since (a friend, who switched from Colab to Delta Drills for ARENA), no formal interviews; the author is the only sustained user. How we find out: the GEAR cohort (if accepted) and the local AI-safety group, asked about their last stuck moment.
Maybe it is better to deploy the skills people already have. Our view: short timelines are why fast upskilling matters. The value is near zero if the window is months, and highest at 2–5 years. This is an open crux, not a settled answer.
Is a missing prerequisite really what stalls people? Untested. Does the engine measure right? On the one learner with data it under-predicts by 13 points (Fig. 7).
Non-profit by design, so there is no capabilities revenue to chase; distributed only through safety networks (GEAR, AISC, local groups); the graph is scoped to the safety curriculum (ARENA). Capabilities labs are not ramp-constrained; the safety field is. No novel ML research is produced.
Every estimate carries an interval; low-evidence concepts show wide bands by construction. Placement output is advice to a learner, never an admissions filter without a human in the loop.
We measure success downstream — ARENA exercises completed, facilitator time saved — not drill streaks. Delta Drills is the prerequisite layer, not a replacement for doing research.
Pilots are small and opt-in; we do the analysis. Colab vs. Delta Drills, same checkpoint tests: if there is no learning gain, we say so publicly and stop.
08 · Project lead
AISC requires at least 10 h/week; the lead commits full-time.
Organized with a group at EA UT Austin; no team-project experience beyond that. This would be the first time leading a research team.
09 · Open questions
This page is written to be critiqued. One question per AISC criterion:
Try it — no account needed. Everything works signed out, and your progress is saved to this browser. Signing in with Google starts an account you can use on any device; what you do as a guest stays in this browser.
You're reviewing the content, not practicing it. Everything you flag lands with a maintainer.
Upload a PDF and optionally a chapters CSV. Use Auto TOC if you want the backend to detect chapters.
Your Delta Drills account.
ARENA takes you from PyTorch fundamentals to original AI-safety research, and it assumes you can already write the code — the tensor manipulation, the einops, the training loops. This app is the practice engine for exactly that assumption. It drills the mechanics behind the chapters below, tracks which ones you have, and sends you to the real ARENA exercise once you're ready for it. It is not a replacement for the course; it is the reps the course expects you to have already done.
You're on the main edition — drills are solved in the app's own code editor. You're on the Colab edition — drills open as Colab notebooks, and the app tutors alongside them.
Follow everyone's goals, daily practice, and competency over time. Start a study group, join a listed one, or paste an invite link.
ready for the ARENA curriculum
Each problem has its own clock, set by its concept — the same one the placement test uses.
Each bar is this area's estimated readiness. Take the placement test from Account and Settings to measure them instead of assuming them.
Problems answered each day, Monday through Sunday.
About this lesson
import torch (sign in to grade PyTorch drills right here), so this one runs in a Colab notebook. Open it (it's the problem — starter code, no answer), work through it there, then tell us how it went. Stuck? Show solution opens the worked answer separately.This reaches the person who maintains the question bank. Say what kind of thing it is, then what happened.
Finds where to start you. It seeds practice; it never awards mastery.
20:00 per problem
Leave out anything you already know well — it won’t be tested, and the test gets shorter.
Sets where the first problems start. Your answers take over from there.
Longer tests pin down more concepts. It stops early once every concept is settled, and never runs past the time you pick.
No areas came back for this placement — retake it to get a fresh reading.