langcurriculum

• 180 lesson generators, not a dataset — an episode is a pure function of a seed, so an evaluation set did not exist until you asked for it and cannot have been trained on
• The answer is computed from the construction, never annotated: the generator invents the grammar, the ontology, the causal graph or the proof calculus, then reads the answer off its own work
• Every score carries a floor measured on the same episodes, so "what does knowing nothing get?" is a number rather than an intuition
• One episode can be asked in 411 languages, five renderings and eight answer formats — and because those are all the same string, a system that answers one and not another has learned the surface
• Zero runtime dependencies. The rasterizer, the animation encoder and the speech synthesizer are all written out, because a dependency would make the corpus a function of somebody else's version

The whole corpus is browsable, and browsing it is the fastest way to tell whether any of the claims below are true. Every lesson has a page, every page carries 50 real episodes, and every episode is shown in all 64 published languages at once:

try it out ↗

Problem

Every claim about a model's reasoning runs into the same two objections, and neither has a good answer.

The first is contamination. A benchmark is a fixed set of questions, it gets published, and thereafter nobody can tell whether a score reflects ability or recall. You can hold data back, but held-back data is still data — it exists, it leaks, and the argument about whether it leaked is unfalsifiable.

The second is subtler and matters more. Suppose a system answers correctly. Did it learn the problem, or the way the problem was written down? A benchmark presents each question exactly one way, so the two are perfectly confounded, and no amount of accuracy separates them.

Solution

Make the questions a program rather than a corpus, and make the presentation a separate axis from the problem.

A lesson is a generator: hand it a seed and it builds a world — a grammar, an ontology, a causal graph — and computes the answer by reading its own construction. There is no labeller to disagree with, and no held-out set to argue about, because the world did not exist until it was asked for. Two different seeds are two different worlds; a million seeds are a million.

The presentation is then peeled off entirely. The same episode can be rendered in any of 411 languages, in five media, in eight different answer formats — and every one of those carries an identical underlying string. That is what makes the confound measurable instead of invisible: render one problem five ways, and ask whether the answers agree. A system that internalized the structure is invariant. One that learned a surface is not, and it shows.

That claim is checkable rather than rhetorical, and this is where to check it. L029 grammar_induction is one episode — seed 000, eleven labelled strings, four candidates — laid out in every published language in a row. The English asks which object is grammatical, the Turkish asks hangi nesne grammatical, the Chinese asks 哪个物体是 grammatical, the Swahili labels its examples ndiyo and hapana, and the answer under every one of them is the identical string rv vl kz. Nothing was translated; the same episode was said 64 times.

try it out ↗

The languages are deliberately not the usual six. Alongside Spanish, German, Japanese and Arabic sit Navajo, Chukchi, Lushootseed, Yahgan, !Xóõ, Nivkh, Ainu, Lakhota, Adyghe, Ingush, Greenlandic, Woiwurrung and Chickasaw — chosen because the point is to break any correspondence between the shape of a sentence and the shape of the problem, and a set of closely related European languages would fail to break it. There is also a Symbolic notation surface and an English (held-out synonyms) one: the first drops natural language entirely, the second keeps English and changes only the words, which isolates lexical memorization from everything else.

How

Lessons are flat; ordering is an opinion. A lesson has no number and no section — it does not know where it sits. Sequence and prerequisites belong to Curriculum objects, directed acyclic graphs over the same lessons, of which several ship and disagree with each other. Only one has edges, and they are derived: X comes before Y when Y is at least as demanding on every declared difficulty axis and strictly more on one. Hand-writing 169 prerequisites would have been inventing structure nobody measured; a derived relation is auditable and wrong in ways someone can point at.

That graph pays for itself immediately. For any node, everything upstream is a training set and the node itself is held out — so the DAG is a generator of 97 compositional-generalization tests, with disjointness by construction rather than by sampling.

Five curricula ship and each one is drawn. canonical is the taught order over all 180; core170 and supplementary cut the same catalogue 170 / 10; everything is the whole set again with nothing withheld; and progressive is the only one carrying edges — the derived DAG, laid out in seven layers of 83 · 39 · 18 · 16 · 15 · 6 · 3.

try it out ↗

The interesting thing in that picture is the first layer. It is 83 lessons wide — nearly half the catalogue — because the derivation rule only draws an edge when one lesson dominates another on every declared difficulty axis and strictly exceeds it on one. Most pairs of lessons are simply incomparable, and a hand-authored curriculum would have hidden that behind a confident ordering.

Difficulty is a dial, not a switch. 177 of the 179 working lessons take a difficulty in [0, 1] that scales whatever actually makes that lesson harder: proof length, recursion depth, the number of rival theories to eliminate, the width of a game to solve. The two that refuse say so honestly — one has exactly one object per compass direction, which is the whole reason each direction denotes uniquely.

Everything is rendered from scratch. The raster surface draws with a bundled 5×7 bitmap font through a scanline fill; the animation is APNG, chosen because it is PNG chunks rather than a codec and therefore byte-exact where an .mp4 drifts between encoder builds; the audio is letter-to-sound rules feeding three formant resonators, the way speech was made before anything was trained. It sounds like 1980, and that is the right trade — what a dictation must preserve is which words were said.

The same episode rasterized with the bundled 5x7 bitmap font: nine objects listed with colours, shapes and coordinates, then the question and its answer options

One surface is deliberately not a re-presentation of the text. scene reads the episode's structure and draws it — the picture is of the scene, not of the sentence describing it — with the question beside it in words. That is visual question answering, and it is kept separate in the code, because a transcode inherits the lesson's floor and a native drawing does not.

The same episode drawn natively: nine coloured shapes on a labelled grid at the coordinates the episode gives them

The catalogue

The 180 lessons are L001 through L180, grouped into 18 tracks that run from "what is a symbol" to "what does a civilization do with one". The arc is the point — each track is a claim about what has to be in place before the next one is a coherent thing to ask.

Each lesson carries its own metadata, and all of it is on the page rather than in a paper: the tags it belongs to, whether it takes a difficulty knob, its position on each declared difficulty axis — grammar complexity, lexical novelty, reasoning depth — and the capabilities it claims to exercise, from finite_state_induction through scientific_induction and abstraction. Those axes are not decoration. They are the input to the prerequisite derivation, which is why the ordering is auditable at all: you can disagree with an edge by pointing at the axis that produced it.

The site publishes 50 sampled episodes per lesson with their computed answers, which is a deliberate and slightly awkward choice — it is a benchmark whose answers are printed next to the questions. It works because the samples are not the evaluation set. They are 50 draws from a generator with an unbounded seed space, published so the corpus can be inspected, and any real evaluation asks for seeds nobody has seen. A fixed benchmark cannot make that trade; this is the one structural advantage of shipping a program instead of a corpus.

Tests

4,026 tests, and the ones that matter are not unit tests.

Every lesson has to pass an admission test before its data may be exported: it generates without exception, the same seed gives the same episode, and a constant guesser scores near chance rather than near one. That last check exists because two generator bugs in this codebase's history produced fake competence — object-id order correlated with the answer, and referring expressions that did not uniquely refer — and both would have looked like a model solving something.

Since replies went open-form this matters more, not less. Nothing grades the output any more, so a lesson whose answer has quietly stopped matching its observation does not show up as a bad score. It shows up as a corpus that teaches the wrong thing, forever. Verification is a gate on export, not a report.

The strongest test in the repo is a negative one. Adding difficulty knobs to 177 lessons had to leave every un-scaled episode byte-identical, so a harness captured all 179 lessons' output first and refused any change that moved it. The proof it worked: regenerating the 125,300 committed sample records afterwards changed one file, the manifest.

Results

180 lessons, 179 of them generating, every one verified to have a beatable floor. 411 languages from a unification grammar with induced morphology, of which 64 are published. Six renderings, 8 answer formats, 44 topical tags, 5 curricula.

Three real bugs were found by instruments rather than by reading. A vocabulary harvest was swallowing a TypeError inside an except Exception and silently dropping four lessons, so their section headings would have rendered in English in every language. The published language extract was missing 163 of the 989 translatable keys, because the probe that built it rendered one language and assumed the key set was universal — a case-marking language never looks up a preposition, so probing Finnish alone never asked for "at", and Spanish shipped with an English "at" in the middle of a Spanish sentence. And a hand-written pack turned out not to be self-sufficient: it falls back to the database under an ISO code the extract had never been told to copy.

What broke when it was used

The curriculum's first real user was symbolic-ai-models, a harness that trained about twenty neural and rule-based architectures on it in August 2026. Ten days of that turned up problems that no admission test here could have caught, because they only show once something is trying hard to score. The curriculum's last commit predates all of them, so every fix or mitigation lives in the harness, not in this repo.

Problems with the episodes themselves

• Rigid templates. The renderer has no randomness, so word order carries structure a model can exploit: 0.54 bits of order structure at depth 128 against −0.05 for natural English. Diagnosed, never fixed; a varied-presentation option was designed and not run.
• Scaffolding swamps the answer. 98.6% of the tokens in an episode are instructions and framing, and 1.4% is the answer. A model can be excellent at next-token prediction and useless at the task (n-gram token lift 0.77, answer lift 0.05), so answer metrics had to be reported separately.
• Node type is almost fixed by the tree's shape. A floor that only looks at shape gets the type right 86% of the time, and five of the seventeen node types never occur at all. Most apparent "type skill" was a lookup table.
• A fixed symbol pool. Every label scored at evaluation had already appeared in training, and attributes such as colour, shape and name are drawn independently. Memorising vocabulary pays, which the renaming test exposed:

• Filling a hidden node is mostly copying. The best model scores 0.492 where the answer is visible elsewhere in the graph and 0.003 where it is not. The task measures retrieval more than generation.
• More data buys memorisation. Raising one model's example budget lifted its accuracy from 0.273 to 0.352, but its accuracy on renamed episodes only from 0.214 to 0.221: about 91% of what the extra data bought was vocabulary.

Problems with using it as a benchmark

• Floors that were too generous or too harsh. The label floor read unmasked parents and the agent floor was fitted differently from the model it judged; both were fixed. An episode floor that counted gold answers from the evaluation episodes themselves was recorded and left as a known bias.
• A slice that ranks backwards. Every early number came from six lessons chosen early and never revisited. On those six, rule induction scored 0.206 below its floor; in the first sweep over all 179 it scored 0.250 above it, and analogy moved from −0.045 to +0.256 the same way. Three per cent of the curriculum ranked the models close to backwards.
• Interactive versions don't discriminate. Turned into environments, nothing cleared its floor except on one task (repair), and a policy that paid to gather information was bit-identical to one that didn't.
• The tooling. The harness's tokenizer deleted underscores, so 9 of 85 answer strings could not be produced at all. A feature built from the same training folds leaked. And an unpinned curriculum version could silently invalidate every result measured on it, which is why the harness now checksums the copy it uses.

The verdict, written on 18 August, was to demote the curriculum from benchmark to development harness. It is very good at producing unlimited, contamination-free problems with known answers. On its own it is a weak judge of whether a system understood them, because its surface is too regular and its vocabulary too small. Both are fixable here (random presentation, an open symbol pool, a shape-blind type distribution), and they are the next thing this generator needs.

Lessons

Measure the boundary; do not declare it. Every coverage question here was answered by instrumenting the thing and rendering real episodes — which characters the font actually needs, which lessons a renderer can actually draw, which keys the grammar actually asks for. The one place that was declared rather than measured is the one place that was wrong for its entire existence, and it was wrong in the most expensive direction: silently, in production, in 57 languages.

A corpus is allowed to have a boundary. It is not allowed to have a silent one. The dictation surface only has English rules; it now says so, and refuses to claim a Spanish episode was read correctly. CJK cannot be drawn by a 5×7 font and is reported missing rather than rendered as boxes. Stating a limit costs a line of code and is the difference between a resource and a liability.

Put the version in the key. A cached rendering addressed without the renderer that made it goes stale the moment the renderer changes, and nothing notices. That exact failure had already happened once here — a published site drifted from its own source for three commits — which is why the cache key names the renderer and the episode key does not.