AGENTIC ROBOTIC MANIPULATION

InteractVLA

Turning VLAs into Interactive Tools for Agentic Robotic Manipulation

Give the agent a target.   Give the policy room to move.

Chen Zhao1,2,3*Zipeng Ye3*Shifeng Bao1,2Yihan Lin1,2Haitao Shen1,2Yining Luo1Yuchen Xie3Jing Zhang1,2†
Renmin University of ChinaMeituan
REASON IN GEOMETRY. ACT WITH A VLA.
THE RESEARCH COMPANION
Paper ↗Supplementary material ↗Code on GitHub ↗

INTERACTVLA IN ACTION

See the interaction.

Browse every task ↗

01 / THE IDEA

A policy the agent
can work with.

A language command says what to do. A geometric target helps the policy understand where and how to act.

When an unfamiliar instruction or a changed scene causes a VLA call to fail, repeating the same command gives the agent little control. InteractVLA opens a two-way interface: the agent steers a pretrained flow-based policy with an end-effector target, then inspects its proposed motion to refine the next action.

Specify a target→Generate motion→Inspect & refine↺

02 / HOW IT WORKS

A two-way conversation
between intent and action.

The agent specifies an interaction pose. A frozen flow-based VLA proposes a motion. The agent inspects that proposal and decides what comes next.

01 · AGENTSpecify a targetPosition · orientation · gripper
02 · INTERACTVLAGuide the motionFrozen policy · test-time guidance
03 · AGENTInspect & refineAction chunks · camera reprojections

↶ Revise the target or adjust the motion. Then continue.

Paper method: soft-time test-time guidance steers the flow denoising process; action-informed agentic refinement inspects and revises motion proposals.
THE COMPLETE FRAMEWORK

Geometric guidance turns a target into input gradients through the action expert. Action-informed refinement returns the generated motion to the agent for the next decision. Enlarge ↗

01Tell it where.
Let it move.

TEST-TIME GEOMETRIC GUIDANCE

Steer a pretrained policy through geometry.

At each tool call, the agent supplies an end-effector target G = (p*, R*, w*): a position, an orientation, and a gripper state. The VLA keeps its learned motor behavior, while guidance redirects its action generation toward that target.

During flow denoising, we estimate the clean action chunk, map its native actions to command poses, and compute a complete-pose error at each execution step. Differentiating that objective through the current action expert produces a correction to the noisy action input. The policy weights stay frozen.

mt = xt + (1 − t) vθ(xt, t; c)
The guidance update
gt = ∇xₜ Lt(C(mt), G)
xt+Δt = sg[xt + Δt (vθ − Bt(gt))]

Bt scales and caps the correction and limits guidance to an interior time window. Each integration state is detached so gradients pass through the current expert only. For bimanual actions, pose errors are summed across effectors at a shared execution index.

Compatibility is Ct = exp[−max(rt/r0, dt/d0)]. After the first guided step, Ct < e−1 triggers early return.

02The right pose,
at the right moment.

SOFT-TIME TARGET ALIGNMENT

Choose when to arrive, while generating how.

A pose target does not specify its arrival time. Pulling every action step toward it can collapse the approach; fixing the last step imposes a timing choice that may not fit the motion. InteractVLA treats arrival time as a latent index in the action chunk.

Early in denoising, several compatible execution steps share the guidance. As denoising progresses, the assignment sharpens around the best-matching step. Position, orientation, and gripper state share the same temporal weights, so the policy aligns a complete interaction pose.

Paper figure showing soft-time guidance focusing progressively on the compatible action step during denoising.
Soft-time alignment preserves the approach while concentrating guidance on a compatible arrival step.
TRY THE IDEA

Watch the temporal weights sharpen.

Drag me →
ApproachCompatible target poseContinue

Conceptual illustration using fixed example pose errors, not a recorded rollout. The actual method recomputes errors as its trajectory changes.

The soft-time objective
τ = 1 − t,   βt = t² / τ²
Lsoft = −τ² log [(1/H) ∑h exp(−βtEh)]
qh(t) = softmaxh(−βtEh)
gt = tτ ∑h qh(t) ∇xₜEh

The guide objective is Lguide = (τ/t)Lsoft. Increasing βt sharpens the temporal assignment; the tτ factor separately modulates guidance amplitude.

03Don't just retry.
Look, then refine.

ACTION-INFORMED AGENTIC REFINEMENT

Make the motion proposal useful to the agent.

A success flag says little about why a motion went wrong. InteractVLA exposes the full action chunk, including the executed prefix and unexecuted continuation, alongside observation history. Calibrated camera reprojections connect predicted command poses to what the agent can see.

With visual observations and proprioception, the agent can compare the approach direction, orientation, and grasp timing, then choose the recovery that matches the problem.

A

Revise the target

If the intended interaction pose is wrong, update G and call the guided VLA again to generate a new, coherent action chunk.

B

Refine the motion

If the maneuver is already useful, adjust an approach offset, lift height, or gripper-release timing through local robot-control tools.

Geometry guides the policy.
The policy’s motion guides the agent’s next decision.

And across attempts?

Turn successful interactions into reusable strategies.

Recursive self-improvement uses practice environments with distinct seeds and scene layouts. The agent consolidates effective target specifications and motion corrections, then transfers those strategies to held-out evaluation instances. This is a separate setting from the online interaction loop above.

03 / FROM INTERACTION TO IMPROVEMENT

Better tools.
Stronger strategies.

With recursive self-improvement (RSI), the agent consolidates effective target specifications and motion corrections, then transfers them to held-out instances.

INTERACTVLA + RSI · OVERALL SUCCESS RATE

99.4%

LIBERO-Plus

Robustness across visual,
language, and spatial shifts.

Main paper · Table 3
48.62%

RoboDojo

Bimanual manipulation and
diverse generalization tasks.

Main paper · Table 4
89.4%

RoboCasa365

Atomic and composite tasks
across kitchen environments.

Main paper · Table 5

Reported with the MR-0 backbone. LIBERO-Plus uses GPT-6.1 Sol; RoboDojo and RoboCasa365 RSI use GPT-6 Astra. These RSI results are distinct from online tool use below.

ONLINE TOOL USE

Geometry helps on the first loop, too.

Language-only Agent + VLA → Agent + InteractVLA
GPT-6.1 Sol + MR-0 · overall success rate

LIBERO-Plus

93.1% → 98.3%

RoboDojo

18.0% → 26.0%

THE EXPERIMENTS

Results, in detail.

Explore the paper and selected published baselines.
Select a metric to change the ranking.

Swipe to compare all metrics ↔
Read the paper ↗

04 / PHYSICAL INTERACTION

Beyond the
simulation.

Tabletop tidying, bottle reorientation, water pouring, and cube placement on the Dexmal platform.

WATER POURING / AGENT+VLA
REAL-WORLD EVALUATION

Reason, manipulate,
and recover.

Geometric targets connect the agent’s intent to the policy’s learned motor behavior. On the paper’s four physical tasks, InteractVLA improves task score over language-only Agent + VLA.

The paper reports normalized task scores, rather than binary success rates, over 30 trials per method and task.

Explore the real-world videos ↗
SUCCESS & EFFICIENCY

Make each interaction count.

Success, token cost, and elapsed time across the three simulated benchmarks. Costs include all RSI attempts, including retries.

Paper figure comparing success rate, normalized token cost, and normalized time for Agent, Agent plus VLA, and Agent plus InteractVLA.
Cost axes are logarithmic and normalized to Agent + VLA. Time includes agent latency, VLA inference and guidance, and environment execution. Main paper, Figure 7.

05 / WATCH IT WORK

The video explorer.

Choose an environment. Pick a task.
See the interaction unfold.

MORE TASKS IN LIBERO-PLUS

Citation

@misc{zhao2026interactvla,
  title = {Turning VLAs into Interactive Tools for Agentic Robotic Manipulation},
  author = {Chen Zhao and Zipeng Ye and Shifeng Bao and Yihan Lin and
            Haitao Shen and Yining Luo and Yuchen Xie and Jing Zhang},
  year = {2026},
  url = {https://openreview.net/forum?id=ytXuX6Zm6J}
}