RRP: relational attention for a structured latent packet

RRP asks one question. What must a robot policy say about its intent if the policy must survive a change of body? The test answer is a latent packet. A planner emits a continuous tensor $z\in\mathbb{R}^{K\times M\times D}$: one slot per knot time and per body assembly. A fast controller on the body turns the received packet into joint targets. It uses only what that body senses. Task meaning must live on $z$. The project requires two kinds of evidence for this: probes, and causal edits of the packet.

The name began as relational robot policy. The repository is now structured-psi0-latent-diffusion-dynamics. All work is in simulation. No physical robot was commanded.

Every clip states its action source on the frame. A scripted teacher and a learned policy look the same in a video; the label is the only difference.

Problem

A policy trained on one arm usually knows nothing useful about the next arm. The interface between "what must happen" and "how this body does it" is implicit. It is buried in weights, and it does not transfer. Also, the network must rediscover structure that the engineer already knows: which joint is the parent of which, what touches what, which object the task is about, which widget a label belongs to. Often it does not rediscover it.

Solution

Two ideas. Each is separate, so each can be tested alone.

1 · The packet interface. The naming follows $\Psi_0$.

• System 1 is an encoder–decoder transformer trained by rectified flow. It reads typed context: morphology, scene, task events and roles, touch. It samples $z$ in 8 Euler steps.
• System 0 runs every control tick. It builds one query per actuated joint from public morphology and the measured state. The queries cross-attend to the packet knots. The output is one normalized joint target per joint.
• System 0 never sees the task, the instruction or an image. Meaning reaches the body only through $z$.
• Packet queries come from public morphology, not from a per-robot lookup. A new body therefore produces new queries.

2 · Relational attention as a sum of declared terms. Every attention call in the model uses one logit:

Each factor $f$ is one entry in one registry: field × operator × form × source. It adds one term $v_f(i,j)$ in one of four forms:

• bias — added to the logit;
• augmentation — appended to $q,k$, so fused attention kernels still apply;
• gate — $\gamma_f=2\sigma(u\cdot c+b)$ scales another term by a task, goal or body summary $c$;
• mask — hard.

Relative geometry follows PaPE. Graph terms come from declared morphology and from the supplied task graph. Some pair terms cannot be read from public inputs: contact, support, held-by, handover. These are bilinear, $\langle Ux_i,Vx_j\rangle$. The bias is then a supervised attention subspace, and its probe estimate is usable at run time. Every coefficient starts at zero. Switching a factor on in a trained model is a no-op at step 0. A deploy guard refuses all simulator-truth sources at run time. Ground truth trains the probes. Only public or estimated values steer attention.

$\mathrm{logit}(i,j)=\frac{⟨q_i,k_j⟩}{\sqrt{d}}$learned · no panel$+$$b_{\mathrm{dist}}$$+$$b_{\mathrm{dir}}$$+$$b_{\mathrm{kin}}$$+$$b_{\mathrm{contact}}$$+$$b_{supp/flow}$$+$$b_{\mathrm{held}}$$+$$g_{\mathrm{task}}{b}_{\mathrm{next}}$$+$$b_{\mathrm{ui}}$

$b_{\mathrm{dist}}$geo.pos3d, distance part: −(r_j − r_i)ᵀM(r_j − r_i) from the cube over every pixel of a Panda pick-and-place state. Hand-set coefficients.

$b_{\mathrm{dir}}$geo.pos3d, direction part: b̃ᵀ(r_j − r_i) from the cube, same state. Hand-set coefficients.

$b_{\mathrm{kin}}$kin.ancestor: the ten ancestors of the G1 left wrist in its declared kinematic tree. Given structure.

$b_{\mathrm{contact}}$ix.contact: gripper–cube contact at grasp close, with the simulator contact points. Simulator label; trains the probe.

$b_{supp/flow}$ix.force_flow: the bottom cube of a three-cube stack carries the two above it. Simulator label closed by the flow operator.

$b_{\mathrm{held}}$ix.held_by: the lifted cube is held by the gripper. Simulator label; trains the probe.

$g_{\mathrm{task}}{b}_{\mathrm{next}}$task.next_contact: the posterior over the gripper's next contact after one distractor is excluded — 0.5, 0.5, 0. Synthetic evidence schedule.

$b_{\mathrm{ui}}$ui.label_for: the Name label points at its text box in a rendered Computerworld form; grey frame is ui.above. Given structure.

How

• Language / runtime: Python 3.12, PyTorch, one rrp CLI. A resource broker runs every heavy job on two NVIDIA GB10 machines.
• Simulation: native MuJoCo with MuJoCo Menagerie robots; a batched Warp legged environment; the $\Psi_0$ SIMPLE benchmark on Isaac Sim; Computerworld as a 3D UI world for a pointer track. 251 distinct bodies ran in at least one recorded run. 150 are in a training set.
• Code and weights: code on GitHub; checkpoints on Hugging Face.
• One interface each for environments, policies and tasks. One harness runs any accepted policy × environment × task. It records each declined pair with its reason.
• Relation-aware data. One privileged state interface feeds four registries: label functions, composable scene parts, factor-agnostic transforms (reveal, surprise, counterfactual swap, noise, occlusion, subsampling) and a composition operator. A scheduler promotes each factor from isolated scenes to composed scenes as its competence rises. The scheduler is implemented and smoke-tested. There is no curriculum result.

The five environments. Arm and dual-arm frames: scripted-teacher states (privileged). Quadruped, humanoid and Computerworld frames: reset states, no controller.

Tests

• Merge gate: pytest tests/unit. It skips, by name, each test that needs weights or third-party assets.
• Routes, on matched seeds: R0 scripted teacher (privileged); R1 oracle packet (diagnostic, not deployable); R2 generated packet (deployable); plus a behaviour-cloning control on the same demonstrations.
• Edit suites change the task context and roll out.
• Semantic and no-semantic variants have matched capacity.
• Splits are sealed and hash-pinned before results. Each sealed cell runs once.

Results

Recorded results only, from the report snapshot of 2 October 2026.

• Semantic supervision helps inside the training bodies. Four arm bodies, two seeds. The latent route with semantic packet supervision succeeds in 425/480 deployable episodes. The capacity-matched control succeeds in 314/480. Difference: +0.23 [0.18, 0.28], ahead in 8/8 body × seed cells. The latent route is still below plain BC on Panda: 135/180 against 58/60.
• The packet is causally used. A "halt" added to System 1's task context can reach the body only through $z$. It cuts forward progress by 0.40 m on anymal_c and 0.66 m on go2 against the control, 3/3 seeds ordered. An irrelevant context change moves the body 0.02 m at most. In Computerworld, a probe-guided packet edit moves the pointer from 163 px to 41 and 22 px of the new target. A random edit of equal norm gives 102–105 px. The edits are partial. Task success does not change.
• Transfer to a new arm is not supported. Zero-shot on sealed xarm7 targets: latent route 0/200, BC 0/200. With target demonstrations, BC fine-tuning reaches 69–192/200. The best latent adaptation was added after the sealed result, so it is post-hoc. It beats a System 0 refit in 12/12 cells and stays below BC in 12/12. The latent route has not beaten BC on any new body.
• Pointer. In distribution: BC 398/400, semantic latent 372, control 368. On unseen words and names, every learned method scores 0/100. None copies characters from the instruction.
• Pretrained humanoid ($\Psi_0$). Released 20/20; direct fine-tune 19/20; structured head 0/20. The cause was an integration fault: System 0 ignored the packet. This is not evidence against structure. The packet-use gate added after the fault passed. The closed-loop rerun is queued.

The report does not claim three things: that the relation factors help (the test is running), that the latent route transfers across bodies, or anything about a real robot. A ten-node pre-registered campaign is in progress: humanoids, arm diversity, $\Psi_0$, the pointer and the relation factors. Its gates were written before the runs.

Lessons

The most useful part of the repository is bookkeeping. An early result said "semantic supervision hurts". It was a confound: an unbounded probe loss made System 0's updates two orders of magnitude smaller. The check found it only because every comparison has a decision number and a capacity-matched control. A source label (teacher, oracle, learned) on every frame and every table costs little. Negative results go in the same ledger as positive ones. Together these keep the project honest about where the latent route is still behind plain BC.

Every factor, one card each

75 implemented relation factors, one deck per family. Swipe a deck left or right, or use the arrows. Each card draws one logit term for one query token (the ring) over a real simulator state. The scene source is in the corner. The factor operators compute the values; they are not learned attention. For a pair term trained from simulator truth, the card shows the label that trains its probe, not the deployed estimate. Three cards are marked illustrative. Tap an image to enlarge it.

Structure

Token-to-assembly membership, slot identity, and which packet knots each System 0 node can read.

edge.node_in_assembly: a joint / sensor token attends to the assembly (arm, gripper) it belongs to

edge.node_in_assembly

a joint / sensor token attends to the assembly (arm, gripper) it belongs to

$1[j=\mathrm{asm}(i)]$

source: public

edge.same_assembly: a command dim attends to the other dims of its own assembly (here the left arm)

edge.same_assembly

a command dim attends to the other dims of its own assembly (here the left arm)

$1[\mathrm{asm}(i)=\mathrm{asm}(j),i\ne j]$

source: public

edge.same_node: an action-node query attends to its OWN morphology token (one per actuated joint)

edge.same_node

an action-node query attends to its OWN morphology token (one per actuated joint)

$1[j=\mathrm{morph}(i)]$

source: public

id.same_assembly: soft same-assembly term: a command dim prefers knots of its own assembly (left arm)

id.same_assembly

soft same-assembly term: a command dim prefers knots of its own assembly (left arm)

$⟨e(a_i),e(a_j)⟩\approx 1[a_i=a_j]$

source: public

id.same_body: tokens denoting the SAME entity (a tracker slot and the task roles bound to it) attend to each other

id.same_body

tokens denoting the SAME entity (a tracker slot and the task roles bound to it) attend to each other

$⟨e({\mathrm{id}}_i),e({\mathrm{id}}_j)⟩\approx 1[{\mathrm{id}}_i={\mathrm{id}}_j]$

source: public

id.slot_handle: each scene token carries its public tracker slot address as a learned embedding, so a slot keeps its identity across steps

id.slot_handle

each scene token carries its public tracker slot address as a learned embedding, so a slot keeps its identity across steps

$x_k\leftarrow x_k+E[{\mathrm{slot}}_k]$

source: public

msg.incidence: role and predicate tokens receive a message from the entity they point to (arrows: entity -> role)

msg.incidence

role and predicate tokens receive a message from the entity they point to (arrows: entity -> role)

$h_i\leftarrow h_i+W{h}_{\mathrm{ptr}(i)}$

source: public

route.assembly_reads: system 0 (Psi0 Realizer): a command dim reads packet knots of its own assembly and its kinematic neighbours only (left arm: arm, hand, torso)

route.assembly_reads

system 0 (Psi0 Realizer): a command dim reads packet knots of its own assembly and its kinematic neighbours only (left arm: arm, hand, torso)

$0\mathrm{if}\mathrm{READS}[\mathrm{asm}(i),\mathrm{asm}(j)]\mathrm{else}-\infty$

source: public

route.own_assembly: system 0 (legged Realizer): a joint node reads only the packet knots of its own limb plus the body assembly

route.own_assembly

system 0 (legged Realizer): a joint node reads only the packet knots of its own limb plus the body assembly

$0\mathrm{if}{a}_j\in \{a_i,\mathrm{body}\}\mathrm{else}-\infty$

source: public

←1 / 9→

Geometry

Relative position, depth, orientation, surface normals. geo.pos3d is the PaPE term in kernel-compatible form.

geo.above: above / below along gravity: +1 for tokens (and surface points) higher than the query by > 1 cm, -1 lower, 0 in the dead zone (query: middle cube of a stack)

geo.above

above / below along gravity: +1 for tokens (and surface points) higher than the query by > 1 cm, -1 lower, 0 in the dead zone (query: middle cube of a stack)

$\mathrm{sign}((r_j-r_i)\cdot \hat{z})\mathrm{if}|\cdot |>1\mathrm{cm}$

source: public

geo.depth3d: PaPE over (u, v, depth) in the declared front camera

geo.depth3d

PaPE over (u, v, depth) in the declared front camera

$-(r_j-r_i{)}^{\top }M(r_j-r_i),r=(u,v,\mathrm{depth})$

source: public + default est-probe

geo.normal_align: query direction b vs each token's contact normal (arrows: outward normal, mean over the token's contacts)

geo.normal_align

query direction b vs each token's contact normal (arrows: outward normal, mean over the token's contacts)

$⟨b_i,n_j⟩,b=(0,0,1)$

source: gt train-only + est-probe

geo.orient: relative rotation between token frames: tr(R_i^T R_j) = 1 + 2 cos(angle) (axes: x red, y green, z blue)

geo.orient

relative rotation between token frames: tr(R_i^T R_j) = 1 + 2 cos(angle) (axes: x red, y green, z blue)

$⟨R_{i}B,R_j{⟩}_F=\mathrm{tr}(R_i^{\top }{R}_j),B=I$

source: gt train-only + est-probe

geo.pos3d: PaPE relative-position term: tokens near and below the gripper score high

geo.pos3d

PaPE relative-position term: tokens near and below the gripper score high

$-(r_j-r_i{)}^{\top }M(r_j-r_i)+{\tilde{b}}^{\top }(r_j-r_i)$

source: public

←1 / 5→

Kinematics

The declared kinematic tree: parents, children, ancestors, siblings, mirror pairs, limbs, and the feet and terrain cells they reach.

edge.foot_of: a limb token attends to its own foot token (feet exist only on foot-carrying assemblies; limbs drawn at hips, feet at foot sites)

edge.foot_of

a limb token attends to its own foot token (feet exist only on foot-carrying assemblies; limbs drawn at hips, feet at foot sites)

$1[j=\mathrm{foot}(i)]$

source: public

edge.limb_adjacent: a limb token attends to the other limbs of the same kind on the trunk (all four legs; tokens drawn at their hips)

edge.limb_adjacent

a limb token attends to the other limbs of the same kind on the trunk (all four legs; tokens drawn at their hips)

$1[\mathrm{kind}(i)=\mathrm{kind}(j)\ne \mathrm{body},i\ne j]$

source: public

edge.over_cell: a foot token attends to the terrain-scan cells of its landing window (0.05 m behind to 0.35 m ahead, +-0.15 m aside of its default stance, body frame)

edge.over_cell

a foot token attends to the terrain-scan cells of its landing window (0.05 m behind to 0.35 m ahead, +-0.15 m aside of its default stance, body frame)

$1[\mathrm{cell}j\in \mathrm{window}(\mathrm{foot}i)]$

source: public

edge.kin_child: a command dim attends to its kinematic children (waist pitch -> both shoulder pitches)

edge.kin_child

a command dim attends to its kinematic children (waist pitch -> both shoulder pitches)

$1[\mathrm{parent}(j)=i]$

source: public

edge.mirror: a command dim attends to its left/right homologue (grey lines: every mirror pair of the body)

edge.mirror

a command dim attends to its left/right homologue (grey lines: every mirror pair of the body)

$1[j=\mathrm{mirror}(i)]$

source: public

edge.kin_parent: a joint token attends to its kinematic parent joint (grey: every parent edge of the four legs)

edge.kin_parent

a joint token attends to its kinematic parent joint (grey: every parent edge of the four legs)

$1[j=\mathrm{parent}(i)]$

source: public

kin.ancestor: a joint token attends to every joint up its kinematic chain (transitive closure of kin_parent; grey arrows: the parent edges)

kin.ancestor

a joint token attends to every joint up its kinematic chain (transitive closure of kin_parent; grey arrows: the parent edges)

$1[j\mathrm{ancestor}\mathrm{of}i]$

source: public

kin.sibling: dims at undirected tree distance 2: the sibling (r_shoulder_pitch) and, by the same op, grandparent and grandchild

kin.sibling

dims at undirected tree distance 2: the sibling (r_shoulder_pitch) and, by the same op, grandparent and grandchild

$1[d_{\mathrm{tree}}(i,j)=2]$

source: public

←1 / 8→

Interaction

Contact, support, force flow, holding, handover. Public inputs cannot give these pair terms. They are bilinear, and each has its own probe.

ix.contact: two entities in geometric contact (sim contact truth): query gripper → cube 1, distractors / target 0

ix.contact

two entities in geometric contact (sim contact truth): query gripper → cube 1, distractors / target 0

$\mathrm{contact}(i,j)$

source: gt train-only + est-probe

ix.force_flow: transitive closure of the support graph: everything the query carries, directly or through others

ix.force_flow

transitive closure of the support graph: everything the query carries, directly or through others

${\mathrm{flow}}^{+}(\mathrm{support})(i,j)$

source: gt train-only + est-probe

ix.handover: two manipulator assemblies in contact with the same object at once (a handover in progress): query L grip → R grip 1, bar / target 0

ix.handover

two manipulator assemblies in contact with the same object at once (a handover in progress): query L grip → R grip 1, bar / target 0

$\mathrm{handover}(i,j)$

source: gt train-only + est-probe

ix.held_by: object held by a manipulator assembly (≥ 2 contacts with its hand bodies): lifted cube → gripper 1

ix.held_by

object held by a manipulator assembly (≥ 2 contacts with its hand bodies): lifted cube → gripper 1

$held\_by(i,j)$

source: gt train-only + est-probe

ix.support: a supports b: contact + contact normal within 30° of gravity-up at a's top

ix.support

a supports b: contact + contact normal within 30° of gravity-up at a's top

$\mathrm{support}(i,j)$

source: gt train-only + est-probe

←1 / 5→

Task

Edges from the supplied task graph — actors, patients, destinations, roles, dependencies, receipts — and the task-gated next contact.

edge.actor_of: manipulator entity ↔ every event it is (cooperating) actor of

edge.actor_of

manipulator entity ↔ every event it is (cooperating) actor of

$1[\{i,j\}\in E_{actor\_of}]$

source: public

edge.consumed_by: runtime receipt ↔ the event whose output-binding role consumes it: offer.anchor is consumed by receive

edge.consumed_by

runtime receipt ↔ the event whose output-binding role consumes it: offer.anchor is consumed by receive

$1[\{i,j\}\in E_{consumed\_by}]$

source: public

edge.destination_of: destination entity ↔ event: the right gripper is where 'offer' and 'release' deliver the bar (target zone is place's destination)

edge.destination_of

destination entity ↔ event: the right gripper is where 'offer' and 'release' deliver the bar (target zone is place's destination)

$1[\{i,j\}\in E_{destination\_of}]$

source: public

edge.enables: event → event it requires COMPLETED (requires_completed): release needs receive and offer done

edge.enables

event → event it requires COMPLETED (requires_completed): release needs receive and offer done

$1[j\in requires\_completed(i)]$

source: public

edge.maintained: event → event it requires ACTIVE (requires_active): receive is only valid while offer is held

edge.maintained

event → event it requires ACTIVE (requires_active): receive is only valid while offer is held

$1[j\in requires\_active(i)]$

source: public

edge.node_actor_of: action node (one joint-command token) → every event its manipulator acts in: a 2-hop composition node → assembly → actor binding

edge.node_actor_of

action node (one joint-command token) → every event its manipulator acts in: a 2-hop composition node → assembly → actor binding

$1[\mathrm{asm}(i)\in \mathrm{actors}(j)]$

source: public

edge.output_to: event ↔ event consuming its output: receive's reference role binds offer's 'anchor' output (event_output binding)

edge.output_to

event ↔ event consuming its output: receive's reference role binds offer's 'anchor' output (event_output binding)

$1[\{i,j\}\in E_{output\_to}]$

source: public

edge.patient_of: patient entity ↔ the events that act on it: the bar is the patient of all five handover events (take, offer, receive, release, place)

edge.patient_of

patient entity ↔ the events that act on it: the bar is the patient of all five handover events (take, offer, receive, release, place)

$1[\{i,j\}\in E_{patient\_of}]$

source: public

edge.pred_arg: predicate-estimate token ↔ its argument entities: held_by(bar, left) points at the bar slot and the left-gripper assembly

edge.pred_arg

predicate-estimate token ↔ its argument entities: held_by(bar, left) points at the bar slot and the left-gripper assembly

$1[\{i,j\}\in E_{pred\_arg}]$

source: public

edge.produced: runtime receipt ↔ the event that produced it: the 'anchor' receipt (interact bank) was emitted by offer

edge.produced

runtime receipt ↔ the event that produced it: the 'anchor' receipt (interact bank) was emitted by offer

$1[\{i,j\}\in E_{\mathrm{produced}}]$

source: public

edge.role_in_event: role-slot token ↔ its event: receive owns four ordered role slots (actor, patient, source, reference)

edge.role_in_event

role-slot token ↔ its event: receive owns four ordered role slots (actor, patient, source, reference)

$1[\{i,j\}\in E_{role\_in\_event}]$

source: public

edge.role_points_to: role-slot token → the entity token it is bound to: receive:source → left-gripper assembly (an output binding points to no entity)

edge.role_points_to

role-slot token → the entity token it is bound to: receive:source → left-gripper assembly (an output binding points to no entity)

$1[j=\mathrm{bound}(i)]$

source: public

edge.support_of: support-role entity ↔ event: the left arm braces the fixture during align

edge.support_of

support-role entity ↔ event: the left arm braces the fixture during align

$1[\{i,j\}\in E_{support\_of}]$

source: publicillustrative: no shipped arm or dual task graph binds a support role to an entity; align:support is rebound to 'left' for this figure only.

edge.target_of: target entity (and every role without its own channel, here receive:source) ↔ event: the left gripper is the source of 'receive'

edge.target_of

target entity (and every role without its own channel, here receive:source) ↔ event: the left gripper is the source of 'receive'

$1[\{i,j\}\in E_{target\_of}]$

source: public

task.next_contact: manipulator → graspable score, sharpened by the task gate (g_task = 1.00 at init, learned)

task.next_contact

manipulator → graspable score, sharpened by the task gate (g_task = 1.00 at init, learned)

$g_{\mathrm{task}}\mathrm{next}(i,j)$

source: candidates + selected: gt train-only + est-probe

←1 / 15→

Locomotion

The stability margin of the centre of mass, and each swinging foot's next foothold.

leg.com_support: readout of the stability margin: signed planar distance of the COM projection to the stance-foot support polygon (m, + inside)

leg.com_support

readout of the stability margin: signed planar distance of the COM projection to the stance-foot support polygon (m, + inside)

signed dist(COM${}_{xy}$, hull(stance feet))

source: gt train-only

leg.foothold: pair label: each swinging foot -> the terrain-scan cell containing its next touchdown (hindsight)

leg.foothold

pair label: each swinging foot -> the terrain-scan cell containing its next touchdown (hindsight)

foothold(foot${}_f$, cell${}_c$)

source: gt train-only + est-probe

←1 / 2→

UI

The Computerworld widget tree: same window, render order, tab order, labels, drag targets.

ui.above: signed render order: +1 if widget j is drawn above i (higher dense z-layer), -1 below, 0 same layer

ui.above

signed render order: +1 if widget j is drawn above i (higher dense z-layer), -1 below, 0 same layer

$\mathrm{sign}(z_j-z_i)$

source: public

ui.drag_to: pair label: the teacher's drag handle -> the nearest widget outside the dragged window to where the drag ends (cw/drag_window)

ui.drag_to

pair label: the teacher's drag handle -> the nearest widget outside the dragged window to where the drag ends (cw/drag_window)

$drag\_to(i,j)$

source: gt train-only + est-probe

ui.focus_next: public UI edge: i, j consecutive in the tab order (focusable, enabled widgets in scene order, cyclic)

ui.focus_next

public UI edge: i, j consecutive in the tab order (focusable, enabled widgets in scene order, cyclic)

$focus\_next(i,j)$

source: public

ui.label_for: public UI edge: a role=label widget -> the next widget after it in the same window (scene order)

ui.label_for

public UI edge: a role=label widget -> the next widget after it in the same window (scene order)

$label\_for(i,j)$

source: public

ui.same_window: 1 when two widgets belong to the same window (parent_id equality; shell items such as the tab strip have parent -1 and never match)

ui.same_window

1 when two widgets belong to the same window (parent_id equality; shell items such as the tab strip have parent -1 and never match)

$1[{\mathrm{parent}}_i={\mathrm{parent}}_j]$

source: public

←1 / 5→

Probes

Readouts in the same registry: what the packet is supervised to carry, read through opaque handles only. Probes are diagnostics. Edits and rollouts show causal use.

probe.arm.acting_on: readout of whether assembly a's hand links touch entity e (contact truth)

probe.arm.acting_on

readout of whether assembly a's hand links touch entity e (contact truth)

$\mathrm{contact}(e,a)$

source: gt train-only

probe.arm.focused_on: readout of which entities a currently ACTIVE task event binds (active op 'place': cube + target)

probe.arm.focused_on

readout of which entities a currently ACTIVE task event binds (active op 'place': cube + target)

$\mathrm{focus}(e)=\exists$ active event $\ni e$

source: public label

probe.arm.goal_effect: readout of the displacement the task asks of each entity: destination minus patient for open events (cube -> target; others 0)

probe.arm.goal_effect

readout of the displacement the task asks of each entity: destination minus patient for open events (cube -> target; others 0)

${\hat{p}}_{\mathrm{dst}}-{\hat{p}}_{\mathrm{patient}}$ (open events)

source: public label

probe.arm.held_by: readout of whether entity e is held by assembly a (here gripper; cube 1, others 0)

probe.arm.held_by

readout of whether entity e is held by assembly a (here gripper; cube 1, others 0)

$\mathrm{held}(e,a)$

source: gt train-only

probe.arm.looking_at: readout of each entity's angle off the front-camera optical axis (gaze label, degrees, scaled 1/30)

probe.arm.looking_at

readout of each entity's angle off the front-camera optical axis (gaze label, degrees, scaled 1/30)

$\angle (\mathrm{cam}\mathrm{axis},p_e-c)$ [deg]

source: gt train-only

probe.arm.observed_effect: readout of each entity's realized displacement over the 16-tick chunk (future_disp; cube lifted, others 0)

probe.arm.observed_effect

readout of each entity's realized displacement over the 16-tick chunk (future_disp; cube lifted, others 0)

$p_e(t+H-1)-p_e(t)$, H=16

source: gt train-only

probe.arm.rel_pos: readout of entity position relative to the assembly's TCP, (x,y,z) m, world frame

probe.arm.rel_pos

readout of entity position relative to the assembly's TCP, (x,y,z) m, world frame

$p_e-p_{\mathrm{tcp}}(a)$ [m]

source: gt train-only

probe.arm.subtask: readout of the active task operator for the assembly (12-way; this episode: grasp -> place -> none)

probe.arm.subtask

readout of the active task operator for the assembly (12-way; this episode: grasp -> place -> none)

$\arg \mathrm{op}(\mathrm{active}\mathrm{event})$

source: public label

probe.arm.visible: readout of whether each entity is visible in the front camera (cube occluded by the closing gripper here: 0)

probe.arm.visible

readout of whether each entity is visible in the front camera (cube occluded by the closing gripper here: 0)

$\mathrm{visible}(e)\in \{0,1\}$

source: gt train-only

probe.legged.contact: readout of each foot's ground contact at the 4 packet knots (+0.1/0.3/0.5/0.7 s) of a trot

probe.legged.contact

readout of each foot's ground contact at the 4 packet knots (+0.1/0.3/0.5/0.7 s) of a trot

$c_f(t+{\tau }_k)$, $\tau$=0.1,0.3,0.5,0.7 s

source: gt train-only

probe.legged.disp: readout of the base displacement over the next 0.8 s in the start body frame (xy in 0.5 m units, yaw change in rad)

probe.legged.disp

readout of the base displacement over the next 0.8 s in the start body frame (xy in 0.5 m units, yaw change in rad)

$R_{{\psi }_t}^{\top }(p_{t+H}-p_t)/0.5,\Delta \psi$

source: gt train-only

probe.legged.fall: readout of an imminent fall: 1 in the last 0.8 s of an episode that ends in a fall (non-foot ground contact, low base or tilt)

probe.legged.fall

readout of an imminent fall: 1 in the last 0.8 s of an episode that ends in a fall (non-foot ground contact, low base or tilt)

$1[\mathrm{fell}]\cdot 1[T_{\mathrm{end}}-t\le 0.8\mathrm{s}]$

source: gt train-only

probe.legged.goal: readout of the active event's goal entity (here the waypoint) in the body yaw frame, /2 m, as the PUBLIC context estimates it

probe.legged.goal

readout of the active event's goal entity (here the waypoint) in the body yaw frame, /2 m, as the PUBLIC context estimates it

goal${}_{\mathrm{active}}$ in body yaw frame / 2 m

source: public label

probe.legged.subtask: readout of which task event is active (walk_to_a, walk_to_b, halt, done) from the public task runtime

probe.legged.subtask

readout of which task event is active (walk_to_a, walk_to_b, halt, done) from the public task runtime

index of the active task event (4-way)

source: public label

probe.pointer.phase: readout of the interaction phase of the commanded tick (idle, move, press, release, drag, type) at each knot

probe.pointer.phase

readout of the interaction phase of the commanded tick (idle, move, press, release, drag, type) at each knot

phase of the commanded tick (6-way)

source: gt train-only

probe.pointer.rel: readout of the target point relative to the pointer, normalized screen units (x right, y up), per knot

probe.pointer.rel

readout of the target point relative to the pointer, normalized screen units (x right, y up), per knot

$x{y}_{\mathrm{target}}(t+j)-x{y}_{\mathrm{ptr}}(t)$ [screen/2]

source: gt train-only

probe.pointer.slot: readout of WHICH widget the current plan step targets, as its stable slot index (opaque codes only, no widget content)

probe.pointer.slot

readout of WHICH widget the current plan step targets, as its stable slot index (opaque codes only, no widget content)

slot index of the step's target widget (80-way)

source: gt train-only

probe.psi0.active_hand: readout of which hand the packet binds to the target (here right)

probe.psi0.active_hand

readout of which hand the packet binds to the target (here right)

hand bound to the target over the packet

source: gt train-only

probe.psi0.base_cmd: readout of the demonstrated base command at each knot (normalized action dims 32 = vx, 34 = turning flag per bodies.g1_simple)

probe.psi0.base_cmd

readout of the demonstrated base command at each knot (normalized action dims 32 = vx, 34 = turning flag per bodies.g1_simple)

demonstrated (vx, turn) command at knot $k$

source: gt train-only

probe.psi0.base_disp: readout of the pelvis displacement over the packet (t -> t+29) in the robot frame at t, plus yaw change

probe.psi0.base_disp

readout of the pelvis displacement over the packet (t -> t+29) in the robot frame at t, plus yaw change

pelvis $(\Delta x,\Delta y,\Delta \psi )$ over 30 ticks

source: gt train-only

probe.psi0.contact: readout of hand-target contact per hand at the 5 knots (here the right hand closes on the target)

probe.psi0.contact

readout of hand-target contact per hand at the 5 knots (here the right hand closes on the target)

$1[{\mathrm{hand}}_h\mathrm{touches}\mathrm{target}](t+k)$

source: gt train-only

probe.psi0.grasp_face: readout of the object face (+x,-x,+y,-y,+z,-z) of each hand's first contact

probe.psi0.grasp_face

readout of the object face (+x,-x,+y,-y,+z,-z) of each hand's first contact

$2\arg \max |p|+[p_{\mathrm{ax}}<0]$ (6 faces)

source: gt train-only (OFF)illustrative: scene real; the label is not computable here (no contact positions in any recorded replay; needs SIMPLE / Isaac Sim).

probe.psi0.grasp_pt: readout of each hand's FIRST contact point on the target in the object frame (last knot)

probe.psi0.grasp_pt

readout of each hand's FIRST contact point on the target in the object frame (last knot)

$R(q{)}^{\top }(p_{\mathrm{contact}}-p_{\mathrm{obj}})$

source: gt train-only (OFF)illustrative: scene real; the label is not computable here (no contact positions in any recorded replay; needs SIMPLE / Isaac Sim).

probe.psi0.hand_dist: readout of each hand's palm-to-target distance at the 5 packet knots (+5..+29 ticks of 20 ms)

probe.psi0.hand_dist

readout of each hand's palm-to-target distance at the 5 packet knots (+5..+29 ticks of 20 ms)

$\Vert p_{\mathrm{palm},\mathrm{h}}(t+k)-p_{\mathrm{target}}(t+k)\Vert$

source: gt train-only

probe.psi0.lift: readout of whether the target is lifted >= 3 cm above its initial height at each knot

probe.psi0.lift

readout of whether the target is lifted >= 3 cm above its initial height at each knot

$1[z_{\mathrm{target}}(t+k)\ge z_0+3\mathrm{cm}]$

source: gt train-only

probe.psi0.target_pos: readout of the target's position at each knot in the robot (pelvis yaw) frame at t

probe.psi0.target_pos

readout of the target's position at each knot in the robot (pelvis yaw) frame at t

$R_{{\psi }_t}^{\top }(p_{\mathrm{target}}(t+k)-p_{\mathrm{pelvis}}(t))$

source: gt train-only

←1 / 26→