canvas-engineering

• Declares latent regions, update clocks, connectivity, and loss roles as a typed schema, then compiles them into attention masks on a frozen pretrained backbone
• Absent edges become exact marginal independence — a hard prohibition in the mask, not a penalty the model can learn to violate
• 26 experiments and 236 training runs on CogVideoX-2B, plus a 23-region cortical model that matches a dense baseline with 19.6% of the connections

Problem

Modern world models denoise a flat bag of tokens. Everything can attend to everything, so whatever structure the model ends up using it has to discover on its own — and once discovered, it lives in weights you cannot read. If you want to know whether a policy's action channel can influence its own observation history, or whether two agents in a fleet are actually coordinating through the channel you intended, there is nothing to point at. You probe, you ablate, you guess.

That is a bad deal, because in most domains worth modeling the causal macrostructure is not the hard part. A nurse's actions obviously affect a patient's next-step physiology, and visual cortex obviously feeds the ventral stream rather than the other way around. Making the model rediscover that from data spends capacity and gives back nothing legible.

Solution

Canvas engineering is prompt engineering for reverse-diffusion latent dynamics: you declare the structure up front and a compiler makes it exact. You lay out named regions on the diffusion canvas, declare which regions are permitted to influence which others, how often each updates, and what role each plays in the loss. The compiler lowers all of that into attention masks, loss weights, and frame mappings on a stock pretrained diffusion transformer.

The division of labor is precise, and it is the whole idea. You fix whether an edge exists, its direction, and its temporal extent. Gradients determine what flows along each edge and everything that happens inside a region. Symbolic is macro, neural stays micro — and there is no interface between the two layers to cross, because the symbolic layer literally is the attention mask. If there is no path between two regions, their independence is exact by construction. You can prove a region cannot influence another by reading the graph.

Familiar architectures fall out as special cases. A standard Markovian diffusion policy is a two-node canvas — observation → action, and observation + action → next observation. Multi-agent coordination is more involved but no less clean: each agent self-attends over its own observations and actions, and agents coordinate only through the cross-edges you declared. The topology is the coordination protocol, written as data.

How

Allocating latent territory by hand doesn't scale past a toy, so I made the library a compiler for typed schema declarations. You write the causal structure as an ordinary Python object — nested typed dataclasses with fields, loss weights, and update clocks — and compile_schema() flattens it into regions and an attention mask.

• Language / runtime: Python, PyTorch; schemas are nested typed dataclasses/pydantic models
• Core API: compile_schema() lowers a schema into a CanvasLayout + CanvasTopology; CanvasModel.from_pretrained() attaches the compiled mask to a stock backbone
• Backbone / data: CogVideoX-2B over BridgeData V2 for the manipulation experiments; Meta's TRIBE v2 brain encoder for the cortical model
• License / install: Apache-2.0, pip install canvas-engineering, with runnable examples for cartpole, vehicle fleets, the hospital model, and brain connectivity

from canvas_engineering import compile_schema, CanvasModel

# Declare the causal macrostructure; the compiler lays it out.
canvas = compile_schema(MySchema)          # regions + attention mask
model = CanvasModel.from_pretrained("cogvideox-2b", canvas=canvas)

model.fit(dataset)                          # gradients learn the weights
state = model.rollout(obs)                  # you fixed the wiring

To stress the compiler, I described the large-scale structure of a hospital ICU ward: six patients with organ-level physiology, four nurses with fatigue dynamics, insurance and staffing pressure, and family units. One compile_schema() call allocated and packed 199 regions and 1,077 connections onto the canvas — heart_rate updating every frame, creatinine every 24, each at its own declared clock.

Every region and every edge is a decision you can read back, which points at a deeper property: the schema parallels C struct semantics. Region bounds are struct offsets, the topology is a calling convention, and a serialized schema is an ABI. The latent tensor becomes pointable — you can name exactly which positions are nurses[1] and read them out. Two models sharing a schema can then exchange latent state directly, with no tokenization or re-encoding round-trip. That's a yawn when the diffusion is just torch on a GPU, and it gets serious the moment you have to repurpose silicon optimized for cat-video generation to do real work.

Results

I ran 26 experiments and 236 training runs on CogVideoX-2B over BridgeData V2 — a small record, and one I would rather report plainly than dress up. The headline is a study of looped attention: iterating frozen diffusion-transformer blocks with learned iteration embeddings. A frozen 3-loop configuration with 350K trainable parameters beat every unfrozen condition at 11.7M: a 1.73× parameter efficiency at p < 0.001, with loop representations converging to fixed points.

Two other measurements keep the story straight. Flat co-residence of modalities — putting every modality in the same tensor with no declared topology — degraded joint prediction by 19% (p < 0.0001); sharing a tensor is not structure. And canvas design turns out to be forgiving: loss was nearly insensitive to exactly how tokens were allocated (α = 0.011). You have to get the graph right, not the pixel budget.

The cortical model is the part I find hardest to look away from. Many neuroscientists have long argued that the cortex's functional connectivity reflects the causal structure of its environment, and the split is familiar — large-scale connectivity is largely specified by the connectome and effectively fixed, while micro-weights are tuned by plasticity over a lifetime. Fixed topology, learned weights. Look closely and a cortical connectivity matrix and a canvas attention mask are the same object: source × destination, block-diagonal within a network, sparse specific cross-network edges, each cell carrying its own operator type.

So I wired 23 Destrieux regions over 42 known pathways — the default-mode loop, the ventral visual stream, A1 → Wernicke → Broca, and prefrontal → premotor → motor — and trained on cortical predictions from Meta's TRIBE v2 brain encoder. The declared-topology canvas saturates at R² = 0.825, matching a fully dense model at R² = 0.826 with 19.6% of the connections and roughly 5× faster training. Topology here is a convergence prior, not a capacity advantage. On virtual-EEG decoding, the canvas decoder hits 69% against an SVM's 59% and 25% chance.

Two scope notes I hold to: the ICU and air-traffic designs are architectural demonstrations, not validated clinical or operational systems; and the cortical model predicts a brain encoder's outputs, not directly measured neural activity. Biology is never as simple as any one model of it — this is a connection worth drawing, not a claim that the cortex is a diffusion transformer.

Lessons

The linchpin I did not get to test is representation stability. Schema-mediated latent exchange, transfer-distance calibration, and plug-and-play modalities all assume that identical declared structure induces predictably aligned geometry across seeds and backbones. Plausible, and unproven — and everything that makes the ABI framing exciting rests on it.

The other honest calibration is about looping. At this scale, its benefit is weight-sharing regularization; there is no detectable iterative reasoning in the results, and reading them as reasoning depth would be reading a 2B-parameter experiment as though it were a frontier-scale one. What is still open, and what I would go after next: a decisive ablation of declared versus flat topology on binding; whether declared structure keeps paying off as data and parameters scale past 2B; and learned topology — proposing and pruning edges under a sparsity prior, with the declared schema as the prior rather than the final word.

The full derivation, the worked designs (manipulation, fleets, air traffic, the ICU, cortex), and the complete experimental archive are in the technical report: Canvas Engineering: Declared Causal Macrostructure for Reverse-Diffusion Latent Dynamics, also published as a CommandAGI research article.