A simple memory layer. A longer view of the task.

SimpleARM

Simple Agentic Memory for
Generalist Robot Policies

Yuyou Zhang1,*, Yunbei Zhang2,*, Miao Li1, Janet Wang2,
Zijian Jin3, Shilong Liu4,5, Ding Zhao1

1CMU · 2Tulane · 3NYU · 4Princeton · 5Columbia

*Equal contribution

Explore the results Paper / arXiv GitHub
SimpleARM illustration: observe an orange mug, maintain its identity, relations, progress and route, then recall and locate the relevant object before acting.
Remember the state that matters for the next action. Conceptual illustration

01 / The idea

Remember state.
Not just frames.

A current image cannot tell a robot which object was hidden, how many actions it has completed, or which route it saw earlier. SimpleARM keeps this task-relevant history available for control.

SimpleARM workflow from the paper: plan which historical state to remember, maintain cube-to-container bindings with frozen perception tools, and recall the relevant binding to ground each pick subgoal for the frozen VLA.
SimpleARM overview. Given the instruction, SimpleARM first specifies what historical state may be needed, then maintains that state online with frozen perception and event tools, and retrieves it only when the current subgoal depends on interaction history. Retrieved memory resolves historical identity, progress, or procedure; current perception provides any required spatial grounding, and the resulting memory-conditioned grounded subgoal is executed by the frozen VLA.
01

Decide what to remember

A vision–language model reads the instruction and proposes the identities, progress and procedures that may matter later.

Instruction → memory plan“Pick the container hiding the green cube.”
Watch the cube–container relation.
02

Maintain it with tools

Frozen perception and event tools verify observations and update compact memory as the robot and its surroundings move.

DetectionIdentityOptical flowMasksEventsRoutes
03

Recall when needed

A history-dependent subgoal triggers the relevant read. Memory resolves the referent; current perception locates it for the frozen policy.

Memory → grounded action“The green cube is under this cover.”
Pick that cover in the current view.

Three forms of task state

Entities & relationsWhich object? Under which cover?
Event progressHow many times? Which step next?
Ordered proceduresWhich route? In what order?

The added memory is episode-scoped and training-free. The pretrained grounded-subgoal composer, robot policy and perception models stay frozen.

02 / Memory in action

Out of sight.
Still the right object.

Follow a real ButtonUnmaskSwap episode. The colors disappear and the covers move, but the memory keeps track of what is underneath.

“First press both buttons on the table, then pick up the container hiding the green cube, finally pick up another container hiding the red cube.”
Step 0: the red and green cubes are visible before being covered.t = 0
Memory-grounded target × Composer proposal
Observe

Notice what matters

The green and red cubes are visible. The instruction selects these identities for memory.

Memory state / readoutGreen cube · red cube

Active operationDetection · color verification

01 / 07
About this example

Seed 7, episode 47, from a qualitative replication of the final configuration. The episode succeeds at step 596. These are recorded front-camera crops; the callouts and memory summaries explain the documented trace. A composer proposal is an intermediate target, not a separate no-memory rollout. Pixel verification, DINO identity features and optical flow were active here; SAM was unavailable on this rollout host.

03 / Across the benchmark

Small memory.
A substantial difference.

Across all 16 RoboMME tasks, SimpleARM reaches 67.17% mean success. The largest gains appear when control needs persistent relations or references to the past.

Overall success rate

Success (%) · one tick = one percentage point

Swipe to see the complete chart →

SimpleARM: mean over 3 seeds. Baselines: published RoboMME results ↗.

Explore all 16 task results
Mean success over 3 seeds (%). Baselines are published results; SimpleARM values come from archived outcomes.
SuiteTaskNo memoryMemERFrameSampSimpleARM

System-level comparison with published baselines. Gains vary by task: Counting remains a limitation, including 0% on StopCube. The added memory receives no training; the pretrained policies use RoboMME’s existing checkpoints.

04 / What makes the difference?

Useful state needs
the right access.

Remove a memory component or change how it is read, then compare with its matched full-method control. Every ablation covers all 800 episodes.

Matched full method vs. ablation

Full method Ablation · success (%)

Swipe to see the complete chart →

800 same-host, same-seed, same-task, same-episode pairs per row · seed 11. Each bead between endpoints = 1 pp of difference.

Maintain the history

Removing route memory lowers full-benchmark success by 11.50 pp; removing demonstration references lowers it by 9.13 pp.

Read it selectively

Replacing structured retrieval with a VLM decision at every proposed subgoal lowers success by 11.88 pp.

Keep the mixed results visible

Changing the source to received frames gives 0.00 pp aggregate change. The table does not show that every component helps every task.

All 10 ablations and exact matched effects
Δ = ablation − matched full method, in percentage points. Each row has 800 pairs.
InterventionFull (%)Ablation (%)Δ (pp)

The matched control varies by row because outcomes are matched to their actual execution host. These are single-seed interventions; component effects overlap and should not be added together.

05 / A closer look at visual history

What if we just keep
the recent frames?

We also fine-tune and evaluate RoboMME’s FrameSamp + ModuL with the same Recent-N strategy at training and test time.

Recent32 has the highest observed mean of these three settings. This is a separate family of trained models from SimpleARM.

Matched Recent training & evaluation

Success (%) · one rung = 1 pp

Whiskers: ±1 sample SD across three evaluation seeds. One training seed (42) and one fixed final checkpoint per setting.

Evaluation details and Recent8

Recent4/16/32 each use evaluation seeds 7, 11 and 23, for 7,200 episodes in total. Some evaluation rounds ran on different hosts, so host effects are not isolated. Recent8 has one completed seed7 evaluation: 211/800 (26.375%). Its evaluated checkpoint used a 70k EMA warm start and reset optimizer; it is not included in the three-seed chart. Earlier inference-only Recent interventions on Uniform-trained weights are a separate experiment.

06 / Conclusion

Remember what matters.
Bring it back to action.

SimpleARM turns interaction history into compact, task-relevant state. A memory plan decides what to retain, frozen tools maintain it as the world changes, and selective retrieval brings it back when a subgoal needs the past.

This simple, training-free memory layer helps frozen robot policies act on identities, progress and procedures that the current image alone cannot reveal.

Task-conditionedTool-maintainedRecalled on demand

Citation

If you find SimpleARM useful, please cite our paper.

@misc{zhang2026simpleagenticmemorygeneralist,
  title={Simple Agentic Memory for Generalist Robot Policies},
  author={Yuyou Zhang and Yunbei Zhang and Miao Li and Janet Wang and Zijian Jin and Shilong Liu and Ding Zhao},
  year={2026},
  eprint={2609.36595},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2609.36595},
}