Pose & reference
Where is my fingertip relative to the key? Which body, object, or world frame gives the instruction meaning?
BODY SYNTONY / A RESEARCH THESIS BY TODD MUSCAT
FIELD STUDY 003 · 2026You are the center
of your own movement.
What if intention had a physical representation? Pose, forces, surfaces, tangents, and normals—expressed through the body’s relationship with its world. From APT’s geometric insight to a proposed interface for learning, prediction, and motion.
Follow the story 01 — 04- Body Syntony /ˈbɑːdi ˈsɪntəni/ noun
- How the physical body aligns, responds, and harmonizes with its movements or its environment.
01 THE FOUNDATION
APT → BODYSYNTONIC
Archive & image source
The foundation: APT.
APT (Automatically Programmed Tools), developed at MIT under Douglas T. Ross in the 1950s, describes motion through relationships with geometry. Imagine yourself as the moving tool: follow a surface, maintain an orientation, stop at a boundary. The system calculates the path. [1] [2]
BodySyntonic builds on that geometric and intent-driven starting point to explore how bodies perceive, predict, and adapt their interactions.
Explore APT’s history, language & interactive examples at VisualAPTRELATIONSHIPS FROM THE BODY’S POINT OF VIEW
02 BODYSYNTONIC / A PROPOSED REPRESENTATION
INTENTION MADE PHYSICALIntention has
geometry.
And dynamics.
Follow a surface. Orient toward its normal. Apply force. Reach a boundary. Adapt as contact changes.
BodySyntonic asks whether these relationships can become a shared representation for learning, reasoning, and motion. BRL—Bodysyntonic Robot Language—is its proposed human-readable expression.
Normals & tangents
Approach along the surface normal. Control lateral drift in the tangent plane. Follow geometry as it moves.
Force & contact
Describe desired interaction: contact mode, force bounds, compliance, friction, and slip.
Time & completion
Specify onset, strike velocity, duration, and release. Define the physical evidence that the action succeeded.
A family of acceptable movements.
An intention describes what should hold while the body moves. Different joint trajectories may satisfy the same relation. A guide surface can be virtual; a contact surface participates in physical interaction. Their roles remain explicit.
Observed state meets desired state.
Keep measured contact, pose, and velocity distinct from intended conditions. Bind every relation to tracked entities, coordinate frames, units, tolerances, and uncertainty. The representation can be a typed graph or tensors, with BRL as a readable view.
03 FROM DESCRIPTION TO PREDICTION
LECUN · WORLD MODELS · INTENTIONWhat do I intend?
What will happen?
A proposed progression of ideas: language, action, predictive world models, and explicit body–world relationships.
01 / DESCRIBE
LLM
Express a task and reason through a symbolic description.
“Play this note.”02 / ACT
VLA
Condition robot actions on visual observations and language.
observation → action03 / PREDICT
JEPA
Learn representations; action-conditioned variants predict consequences.
state + action → future04 / STRUCTURE INTENT
BodySyntonic
Represent the bodily and physical relationships a task requires.
relations → objectivesLLM → VLA → JEPA → BodySyntonic is our conceptual progression, not a historical succession or replacement claim. VLA policies and JEPA world models can work together; BodySyntonic is a proposed interface across them.
The connection to LeCun.
LeCun’s autonomous-agent proposal separates prediction of the world from evaluation of outcomes and selection of actions. We use “intention” here for explicit desired relationships and completion criteria—a candidate input to that evaluation and planning process. [5]
V-JEPA 2-AC provides a concrete reference: it predicts future representations conditioned on actions and proprioception, and supports planning toward image goals. [6]
Where BodySyntonic could contribute.
Score predicted trajectories against contact, pose, timing, and force objectives. Train additional readouts for control-relevant relations. Condition an action proposer on the same intention representation.
A “meta-transformer” could learn to translate observations and a task into this structured intention, or condition candidate actions on it. The first prototype can use an explicit parser and planner; the transformer is a later, testable architecture choice.
GROUND
Sense the body, key, contact, and uncertainty.
REPRESENT
Bind intention to entities and physical quantities.
PREDICT & EVALUATE
Roll out actions and score the relationships.
ACT & OBSERVE AGAIN
Execute a short segment and update the state.
A visual embedding does not automatically provide contact force or a calibrated surface normal. Those quantities need geometric estimation, sensors, simulation labels, or trained readouts. The learned model and the physical measurements must be connected explicitly.
04 FIRST EXPERIMENT / ONE PIANO KEY
SMALL TASK · RICH PHYSICSOne key.
A complete intention.
Locate it. Approach. Establish contact. Strike with controlled timing and velocity. Release. Observe the note.
BSRL / CONTACT LAB
Arrange the scene. Run the interaction.
Move or rotate the keyboard, choose a contact patch and approach angle, then run the motion. The Parent/Hand approaches first; MCP flexion performs the strike while the Hand stays fixed. Requested MIDI velocity controls the scripted strike speed and contact dwell.
What runs here: the latest three.js geometric proof of concept, including its four slides and proposed execution loop. Prediction, online learning, calibrated dynamics, and hardware MIDI are research steps; this animation does not implement them.
FROM THE DEMONSTRATION TO A LEARNED SYSTEM
The same relationships, three roles.
The proposal is to organize a learned world model around body–environment relationships. BSRL (Body Syntonic Robot Language) expresses intention in that same representation: the Hand, MCP joint, fingertip patch, key surface, and hinge form a graph of attachments, articulation, proximity, and contact.
What is happening?
Estimate relative poses, joint state, contact, and uncertainty from observations. Combine explicit geometry with learned sensory features and interaction history.
What should happen?
Express the desired approach angle, contact patch, stable Hand, MCP strike, timing, and MIDI outcome as relationships and phase conditions.
What may happen next?
Condition a predictor on the observed state and candidate actions. Predict future features, relationships, and note events; compare them with the intention.
Observed state and desired state remain distinct. Asking to maintain contact does not mean the model may assume contact succeeds.
Multimodal
Vision locates the key; proprioception describes actual motion; MIDI reports note events and velocity. Optional audio or touch can help resolve ambiguous interactions. Modality encoders feed a shared body-and-interaction graph.
Multilevel
Separate the musical task, the approach/contact/strike/release phases, and joint-level execution. A graph transformer with temporal memory is a candidate architecture; a JEPA-style objective learns to predict future representations. [13] [14]
Learning as you go
Infer the current key response, update a small dynamics or MIDI-outcome model between strikes, then consolidate new and retained experiences. For example, a target of 70 and an observed 52 would inform the next strike. Context inference and weight updates are distinct adaptation mechanisms. [16]
01 / BODY-RELATIVE GEOMETRY
Represent the relationship.
W is the world frame, B the Body frame, and K the key frame. Expressing both poses in another global coordinate frame leaves this relative transform unchanged.
The opportunity is to remove irrelevant world-frame variation and share equivalent interactions. A coordinate change alone does not reduce dimensionality. Gravity, joint configuration, reach, and obstacles still matter.
02 / PROPOSED LEARNING OBJECTIVE
Predict features and consequences.
Combine future-feature prediction with relational accuracy, known mechanics, note/MIDI outcomes, and consistency under valid coordinate transformations. The λ terms weight these objectives; this is a proposed loss, not a trained result.
Use known mechanics; learn what is uncertain. Keep forward kinematics and hinge geometry explicit. Learn residual compliance, damping, and actuator lag, with separate free-motion, contact, and release modes. Physics-informed learning can constrain the model without requiring it to rediscover the mechanism. [15]
LLMs, VLAs, and JEPA supply different building blocks. This proposed combination uses body-relative structure across perception, intention, and prediction; the cited research supports those ingredients, not a demonstrated BSRL system.
Begin with an instrumented key.
Our proposed bench rig uses a single actuator, a compliant fingertip, an encoder over one weighted MIDI key. MIDI reports note onset, release, and velocity. MIDI velocity is the resultant measure of strike intensity for this experiment; the encoder tracks key travel. Add a camera and lateral motion when testing adaptation to key placement.
Measure the musical outcome.
Calibrate the relationship between the strike trajectory and the key’s reported velocity. Use MIDI velocity as the target for musical dynamics, with the instrument’s velocity-to-loudness response kept consistent. Track onset error, MIDI velocity error, unwanted retriggers, and complete release.
| Layer | Build with current tools | Add BodySyntonic |
|---|---|---|
| Bench & simulation | Instrumented key and actuator. Python + MuJoCo for a calibrated hinge, spring, damping, and contact model. | The same rig and physics. Attach semantic roles: fingertip, guide, contact patch, normal, tangent. |
| State & data | Synchronize encoder, MIDI, and optional camera samples. Store state/action trajectories. | Also record relation estimates, reference frames, intention, phase, and uncertainty. |
| Task execution | C++/MCU phase controller: approach → contact → strike → release. Tune trajectory and travel bounds. | BSRL parser → typed intention graph → trajectory objectives and phase transitions. |
| Learned policy option | LeRobot + PyTorch: demonstrations → ACT or a compatible VLA policy → hardware adapter. Fit the observation and action spaces to the rig. | Condition the action proposer on the intention graph. Train relation-aware readouts alongside the learned representation. |
| Predictive option | Train a compact action-conditioned dynamics model. V-JEPA 2-AC is a research reference; a single-key rig needs its own action/sensor adaptation and data. | Predict candidate outcomes and score guide, pose, timing, MIDI velocity, and release objectives in a receding-horizon planner. |
| Deployment | Host process for policy/planning; device-side feedback loop for motor control and travel limits. | The intention runtime provides bounded setpoints and objectives to the same feedback controller. |
The engineered controller is the initial baseline; learned policy and predictive planning are separate comparison paths. Available tools are linked in [8]. No foundation model is required to establish the first repeatable key strike.
RESEARCH HYPOTHESIS
a shared body-syntonic representation for perception, intention and prediction can reduce the experience needed to transfer an interaction and adapt its execution.
Test transfer. Hold out keyboard placements. Compare world-frame state, body-relative state, and the interaction graph with matched observations, model capacity, control, and training budgets. Measure note success, prediction error, and experience required.
Test adaptation. Change key response or actuator delay. Compare a frozen model, context adaptation, incremental learning, and a classical adaptive controller. Measure strikes needed to recover MIDI accuracy, timing, and stable Hand behavior.
These are proposed experiments; there are no benchmark results yet. MIDI velocity is an instrument outcome, not a direct force measurement. Report failures and retain gravity, reach, and joint constraints when testing transfer.
FROM ONE CONTACT TO COORDINATED MOTION
A key today.
A tango
tomorrow.
A larger research horizon: many contacts, moving partners, changing support, and shared intention.
A RESEARCH DIRECTION BY TODD MUSCAT
Intention.
Prediction.
Embodiment.
Body syntonic principles from APT, provide the ego-centric geometric and intent-driven starting point.
Work coauthored by LeCun explores shaping JEPA representations for goal-reaching value. BodySyntonic asks how explicit physical intentions could make goals and predicted outcomes more useful for motion development. [10]
The hypothesis is practical: a shared representation of body–world relationships could connect what we ask, what a model predicts, and what a robot actually does.
Sources, history & image credits+
- Douglas Ross’s 1970 N/C World interview · MIT archive. Also: MIT Science Reporter: Automatically Programmed Tools (1959).
- APT Open Source: What is APT? · Language and toolpath overview. Also: A Description of the APT Language, Communications of the ACM (1963). Examples on this page are teaching fragments, not production machining programs.
- National Bureau of Standards: Standards for Computer Aided Manufacturing · APT entry records first publication in June 1974.
- IronMind: Camera-Space Ego-Centric Pretraining · Research preprint, September 30, 2026.
- Yann LeCun: A Path Towards Autonomous Machine Intelligence (2022) · World-model, cost, and action-selection architecture.
- V-JEPA 2: Understanding, Prediction and Planning · Action-conditioned world modeling and image-goal planning. Official PyTorch implementation.
- π₀: A Vision-Language-Action Flow Model for General Robot Control · A concrete VLA approach.
- LeRobot policy deployment; MuJoCo contact dynamics; Franka Control Interface. These are available building blocks; the piano workflow is a proposed engineering design.
- ReKep: Relational Keypoint Constraints · Related work in representing and optimizing manipulation constraints.
- Value-guided action planning with JEPA world models · Destrade et al., including Yann LeCun. Shaping representations for goal-reaching.
- Historical machine photograph: MIT 2.810 machining lecture (2019), PDF p. 86, credited to Reintjes, Numerical Control (1991). Photograph date unspecified; no open reuse license stated.
- Robot tango: original AI-generated concept illustration. It depicts the research vision; it is not a demonstration of an implemented BRL system.
- Dawid & LeCun: Introduction to Latent Variable Energy-Based Models (2023) · Joint-embedding prediction and hierarchical world-model concepts.
- Body Transformer (2024) · Body-graph structure and masked attention for policy learning. Extending this to body–environment contact prediction is our proposal.
- PIN-WM (RSS 2025) · Physics-informed world modeling and physical-parameter identification for manipulation; a reference for the proposed hybrid dynamics approach.
- DayDreamer (CoRL 2022 / PMLR 2023) · Online world-model learning on physical robots; a reference for learning from ongoing interaction.