Generalizable Priors for Robot Manipulation

Overview

Generalist robot policies are those capable of performing complex manipulation tasks across a wide range of environments. Recent years have seen significant progress toward this goal, driven by advances in large-scale teleoperated robot datasets, structured representations for policy learning, hierarchical planning with vision-language models (VLMs), and online learning for adaptation to novel tasks and environments. Although multiple promising approaches are emerging, it is essential to understand the tradeoffs inherent in each to develop methods that not only generalize to new scenarios but also execute tasks with high precision and reliability.

An ideal framework for learning such generalizable policies should: (1) scale beyond simple pick-and-place tasks and continually improve as more task data becomes available, (2) learn effectively from diverse data sources, including robot teleoperation, simulated environments, and human demonstration videos, and (3) utilize representations of the world that are applicable across tasks requiring varying levels of precision and dexterity. This workshop will focus on a central question: What are the right priors for generalizable policy learning, and how can we best incorporate these priors into policy learning frameworks?

Our speakers and panelists are leading researchers in robotics and machine learning, working at the forefront of topics including end-to-end control, sim-to-real transfer, learning from human videos, and large-scale robotic data collection, among others. We invite the community to submit their latest work and ideas for discussion.

Areas of Interest

We aim to investigate the following topics and research questions:

• What main bottlenecks prevent current robot policies from generalizing to unseen environments?
  • Is the challenge primarily a data problem, or are there fundamental limitations in current approaches?
• Can pretrained models from related fields, such as vision and language, be leveraged to improve robot policies?
  • What is the most effective way to incorporate these models into robotics?
  • Should we fine-tune them on robot data, or use their representations and outputs for downstream policy learning?
• How can we best utilize simulated environments to promote generalization?
  • Do simulated environments need to closely resemble the real world for successful policy transfer, or can they be substantially different?
  • What role do generative models play in the sim-to-real pipeline? Are they consistent enough to make reliable long-term predictions over full trajectories?
• Is it possible to design universal priors that work across a wide range of tasks and domains, including simulation, the real world, and internet videos?
  • Can object-centric representations provide this capability?
  • Can 3D representations facilitate cross-domain transfer?
• How should we design standardized benchmarks for evaluating such generalist policies?

Submission Guidelines

Submission Portal: OpenReview

We are accepting workshop submissions of the following types

• Papers - 4-8 pages plus unlimited references / appendices

We request that submissions are in the CoRL format. They should not be anonymized.