TensorCode

presentation slides

Problem

Building a multi-modal model means writing the same plumbing over and over: pick an encoder per input type, concatenate latents, run a decision network, then hand-decode the output tensor back into the Python objects the rest of your program expects. Every time the Observation or Action shape changes, that glue code changes with it. You end up thinking in tensors when you'd rather be thinking in objects, patterns, and architecture.

Solution

TensorCode introduces simple abstractions and functions for encoding and decoding arbitrary Python objects, differentiable programming (including differentiable control flow), and intelligent object creation, selection, and other runtime code-generation features.

Rather than just think about the underlying mathematical objects, TensorCode's Programming 2.0 paradigm gives your brain the abstractions it needs to apply ordinary software-engineering patterns — encapsulation, abstraction, composition, design patterns, and the concepts derived from them — to your underlying problem. If you're developing a multi-modal agent, building the next generation of end-to-end differentiable cognitive architectures, or just adding magic to an existing program, TensorCode may fit right into your stack.

import tensorcode as tc

def step(self, obs: Observation) -> Action:
    latent = tc.encode(obs)
    action = self.mlp(latent)
    return tc.decode(action, Action)

tc.encode turns any annotated object into a latent tensor; tc.decode turns a tensor back into a typed object. When the Observation or Action definition grows a new field, the encode and decode calls don't change — TensorCode infers the wiring from the type annotations.

How

The framework centers on a few primitives:

• tc.encode / tc.decode — bidirectional conversion between arbitrary Python objects and tensors, driven by type annotations (Image, str, float, Union, Literal, tuples, nested classes).
• Runtime code generation — tc.exec, tc.select, tc.create, and tc.patterns.Factory let a program describe what it wants in natural language and have TensorCode produce, choose, or run the object at runtime.
• Differentiable programming — tc.If / .Elif / .Else decorators make control flow differentiable, so gradients flow through branches the same way they flow through arithmetic.
• Cognitive-architecture building blocks — multi-modal fusion, short- and long-term memory, a global-workspace prediction loop, and composable loss terms for training the whole thing end to end.

Presentation

The deck below walks from a hand-wired RL agent to the same agent expressed with TensorCode primitives, then through runtime code generation, differentiable control flow, and a full differentiable cognitive architecture. The full deck is also available as a downloadable PDF.

Status

TensorCode is an active, exploratory framework. The GitHub repository holds the current implementation. The project notebook tracks design notes. Encoding/decoding and runtime code generation are the furthest along; differentiable control flow and reinforcement programming are still in progress.

Lessons

The recurring lesson is that the bottleneck in multi-modal ML isn't the math — it's the abstractions. Giving the encode/decode boundary a name, and letting type annotations carry the wiring, is what makes the software-engineering toolkit (composition, design patterns, encapsulation) usable on a learning system at all.