Robots are getting better at doing chores, but they still stumble on long, complicated jobs. A new method called SeeQ gives them a smarter way to pick their next action, one small step at a time.
The problem: big jobs are hard to judge
Imagine a robot asked to tidy a table, fold a towel, or pack a box. These jobs have many stages, and sometimes the robot must try the same stage several times before it works. Today’s general-purpose robot “policies” (the programs that decide what to do) are often brittle here. One small slip, and the whole task falls apart.
One fix is to add a “scorer” that rates possible actions, a bit like a coach saying, “That move looks promising, that one doesn’t.” In AI, this scorer is called a Q-value function.
The catch is that most scorers only learn from one signal: did the whole task succeed in the end? That is a lot like trying to learn chess only from knowing whether you won, without feedback on any individual move. It takes huge amounts of data, and the learning process becomes slow and unstable.
The idea: score the current step, not the whole job
SeeQ (short for Subtask-elicited Q-functions) changes the question. Instead of asking, “Will this action eventually finish the entire task?”, it asks, “Will this action finish the step I’m doing right now?”
For a towel-folding robot, the steps might be “grab the corner,” “fold in half,” and “smooth it flat.” Judging one step is much easier, because the outcome comes quickly. The learning is faster and works with messier, less perfect data.
Where the step labels come from
Robot training data often already includes notes about which step is happening. SeeQ uses these during training, so it can learn from large collections of robot demonstrations, even ones where the robot wasn’t doing a great job.
At test time, nobody wants to label steps by hand. So SeeQ is trained to first say, in plain language, what step it thinks it is in, such as “pick up the cup,” and then to rate how well an action fits that step. No human helper or extra system is needed while the robot works.
How it was built and tested
The team started with a vision-language model, an AI that understands both images and text. They trained it on a wide mix of open-source robot manipulation data, then fine-tuned it for specific jobs.
They tested it on four real-world tasks using two two-armed robot platforms. The method used was “best-of-N steering”: the robot’s policy proposes several possible actions, and SeeQ picks the one it rates highest. This significantly improved the robots’ results.
Why it matters
The big takeaway is that breaking a long job into small, judgeable pieces makes robot learning simpler and more reliable. It also lets researchers use the huge, imperfect datasets that already exist, rather than collecting perfect demonstrations.
There are limits. The tests covered just four tasks, and real homes and workplaces are far messier. Still, SeeQ shows that teaching a robot to think “what am I doing right now, and is this move helping?” may be a practical path to more dependable helpers.
Original abstract:
Despite rapid progress, generalist robot policies remain brittle on complex, long-horizon tasks that comprise multiple stages or require repeated attempts and deliberation on the same underlying stage before success. Q-value functions can improve these policies by ranking candidate actions or guiding policy improvement, but learning from sparse task-level rewards entails long credit-assignment horizons, difficult Bellman backups, and broad data-coverage requirements. We introduce SeeQ (Subtask-elicited Q-functions), which instead learns Q-values for the currently active subtask. This shortens the value-prediction horizon and enables effective learning with temporal-difference (TD) objectives. During training, subtask-level annotations present in offline robot data provide the decomposition and enable learning from broad, potentially suboptimal robot datasets. To eliminate the need for human annotations or modular subtask prediction systems at test time, our Q-function architecture is trained to autoregressively predict the active subtask in natural language before estimating its value. We instantiate SeeQ using a base vision-language backbone, pretrain it on diverse open-source robot manipulation data, and finetune it on downstream tasks. Across four real-world manipulation tasks on two bimanual robot platforms, the SeeQ value function substantially improves best-of-N policy steering.
Source: arXiv
Transparency Note: This article was prepared with the assistance of artificial intelligence technologies. Every piece is thoroughly reviewed and verified by our editorial team, who retain full responsibility for the final content.
Picture: Gemini AI