Helping robots zero in on the objects that matter | MIT News

Imagine having to straighten up a messy kitchen, starting with a counter suffering from sauce packets. In case your goal is to wipe the counter clean, you would possibly sweep up the packets as a gaggle. If, nonetheless, you desired to first select the mustard packets before throwing the remaining away, you’d sort more discriminately, by sauce type. And if, among the many mustards, you had a hankering for Grey Poupon, finding this specific brand would entail a more careful search.

MIT engineers have developed a way that permits robots to make similarly intuitive, task-relevant decisions.

The team’s recent approach, named Clio, enables a robot to discover the parts of a scene that matter, given the tasks at hand. With Clio, a robot takes in an inventory of tasks described in natural language and, based on those tasks, it then determines the extent of granularity required to interpret its surroundings and “remember” only the parts of a scene which are relevant.

In real experiments starting from a cluttered cubicle to a five-story constructing on MIT’s campus, the team used Clio to mechanically segment a scene at different levels of granularity, based on a set of tasks laid out in natural-language prompts akin to “move rack of magazines” and “get first aid kit.”

The team also ran Clio in real-time on a quadruped robot. Because the robot explored an office constructing, Clio identified and mapped only those parts of the scene that related to the robot’s tasks (akin to retrieving a dog toy while ignoring piles of office supplies), allowing the robot to understand the objects of interest.

Clio is known as after the Greek muse of history, for its ability to discover and remember only the weather that matter for a given task. The researchers envision that Clio can be useful in lots of situations and environments by which a robot would must quickly survey and make sense of its surroundings within the context of its given task.

“Search and rescue is the motivating application for this work, but Clio may power domestic robots and robots working on a factory floor alongside humans,” says Luca Carlone, associate professor in MIT’s Department of Aeronautics and Astronautics (AeroAstro), principal investigator within the Laboratory for Information and Decision Systems (LIDS), and director of the MIT SPARK Laboratory. “It’s really about helping the robot understand the environment and what it has to recollect with a purpose to perform its mission.”

The team details their leads to a study appearing today within the journal Robotics and Automation Letters. Carlone’s co-authors include members of the SPARK Lab: Dominic Maggio, Yun Chang, Nathan Hughes, and Lukas Schmid; and members of MIT Lincoln Laboratory: Matthew Trang, Dan Griffith, Carlyn Dougherty, and Eric Cristofalo.

Open fields

Huge advances within the fields of computer vision and natural language processing have enabled robots to discover objects of their surroundings. But until recently, robots were only capable of achieve this in “closed-set” scenarios, where they’re programmed to work in a rigorously curated and controlled environment, with a finite variety of objects that the robot has been pretrained to acknowledge.

Lately, researchers have taken a more “open” approach to enable robots to acknowledge objects in additional realistic settings. In the sphere of open-set recognition, researchers have leveraged deep-learning tools to construct neural networks that may process billions of images from the web, together with each image’s associated text (akin to a friend’s Facebook picture of a dog, captioned “Meet my recent puppy!”).

From hundreds of thousands of image-text pairs, a neural network learns from, then identifies, those segments in a scene which are characteristic of certain terms, akin to a dog. A robot can then apply that neural network to identify a dog in a very recent scene.

But a challenge still stays as to how one can parse a scene in a useful way that’s relevant for a selected task.

“Typical methods will pick some arbitrary, fixed level of granularity for determining how one can fuse segments of a scene into what you’ll be able to consider as one ‘object,’” Maggio says. “Nonetheless, the granularity of what you call an ‘object’ is definitely related to what the robot has to do. If that granularity is fixed without considering the tasks, then the robot may find yourself with a map that isn’t useful for its tasks.”

Information bottleneck

With Clio, the MIT team aimed to enable robots to interpret their surroundings with a level of granularity that will be mechanically tuned to the tasks at hand.

As an example, given a task of moving a stack of books to a shelf, the robot should give you the chance to  determine that all the stack of books is the task-relevant object. Likewise, if the duty were to maneuver only the green book from the remaining of the stack, the robot should distinguish the green book as a single goal object and disrespect the remaining of the scene — including the opposite books within the stack.

The team’s approach combines state-of-the-art computer vision and huge language models comprising neural networks that make connections amongst hundreds of thousands of open-source images and semantic text. Additionally they incorporate mapping tools that mechanically split a picture into many small segments, which will be fed into the neural network to find out if certain segments are semantically similar. The researchers then leverage an idea from classic information theory called the “information bottleneck,” which they use to compress numerous image segments in a way that picks out and stores segments which are semantically most relevant to a given task.

“For instance, say there’s a pile of books within the scene and my task is simply to get the green book. In that case we push all this information concerning the scene through this bottleneck and find yourself with a cluster of segments that represent the green book,” Maggio explains. “All the opposite segments that are usually not relevant just get grouped in a cluster which we will simply remove. And we’re left with an object at the fitting granularity that is required to support my task.”

The researchers demonstrated Clio in several real-world environments.

“What we thought can be a extremely no-nonsense experiment can be to run Clio in my apartment, where I didn’t do any cleansing beforehand,” Maggio says.

The team drew up an inventory of natural-language tasks, akin to “move pile of garments” after which applied Clio to photographs of Maggio’s cluttered apartment. In these cases, Clio was capable of quickly segment scenes of the apartment and feed the segments through the Information Bottleneck algorithm to discover those segments that made up the pile of garments.

Additionally they ran Clio on Boston Dynamic’s quadruped robot, Spot. They gave the robot an inventory of tasks to finish, and because the robot explored and mapped the within an office constructing, Clio ran in real-time on an on-board computer mounted to Spot, to select segments within the mapped scenes that visually relate to the given task. The tactic generated an overlaying map showing just the goal objects, which the robot then used to approach the identified objects and physically complete the duty.

“Running Clio in real-time was a giant accomplishment for the team,” Maggio says. “Lots of prior work can take several hours to run.”

Going forward, the team plans to adapt Clio to give you the chance to handle higher-level tasks and construct upon recent advances in photorealistic visual scene representations.

“We’re still giving Clio tasks which are somewhat specific, like ‘find deck of cards,’” Maggio says. “For search and rescue, you could give it more high-level tasks, like ‘find survivors,’ or ‘get power back on.’ So, we wish to get to a more human-level understanding of how one can accomplish more complex tasks.”

This research was supported, partially, by the U.S. National Science Foundation, the Swiss National Science Foundation, MIT Lincoln Laboratory, the U.S. Office of Naval Research, and the U.S. Army Research Lab Distributed and Collaborative Intelligent Systems and Technology Collaborative Research Alliance.