The electricity to an island goes out. To seek out the break within the underwater power cable, a ship pulls up your complete line or deploys remotely operated vehicles (ROVs) to traverse the road. But what if an autonomous underwater vehicle (AUV) could map the road and pinpoint the placement of the fault for a diver to repair?
Such underwater human-robot teaming is the main target of an MIT Lincoln Laboratory project funded through an internally administered R&D portfolio on autonomous systems and carried out by the Advanced Undersea Systems and Technology Group. The project seeks to leverage the respective strengths of humans and robots to optimize maritime missions for the U.S. military, including critical infrastructure inspection and repair, search and rescue, harbor entry, and countermine operations.
“Divers and AUVs generally don’t team in any respect underwater,” says principal investigator Madeline Miller. “Underwater missions requiring humans typically accomplish that because they involve some form of manipulation a robot cannot do, like repairing infrastructure or deactivating a mine. Even ROVs are difficult to work with underwater in very expert manipulation tasks since the manipulators themselves aren’t agile enough.”
Beyond their superior dexterity, humans excel at recognizing objects underwater. But humans working underwater cannot perform complex computations or move in a short time, especially in the event that they are carrying heavy equipment; robots have an edge over humans in processing power, high-speed mobility, and endurance. To mix these strengths, Miller and her team are developing hardware and algorithms for underwater navigation and perception — two key capabilities for effective human-robot teaming.
As Miller explains, divers may only have a compass and fin-kick counts to guide them. With few landmarks and potentially murky conditions attributable to a scarcity of sunshine at depth or the presence of biological matter within the water column, they will easily develop into disoriented and lost. For robots to assist divers navigate, they should perceive their environment. Nevertheless, within the presence of darkness and turbidity, optical sensors (cameras) cannot generate images, while acoustic sensors (sonar) generate images that lack color and only show the shapes and shadows of objects within the scene. The historical lack of huge, labeled sonar image datasets has hindered training of underwater perception algorithms. Even when data were available, the dynamic ocean can obscure the true nature of objects, confusing artificial intelligence. For example, a downed aircraft broken into multiple pieces, or a tire covered in an overgrowth of mussels, may not resemble an aircraft or tire, respectively.
“Ultimately, we wish to plan solutions for navigation and perception in expeditionary environments,” Miller says. “For the missions we’re fascinated with, there is restricted or no opportunity to map out the realm upfront. For the harbor entry mission, possibly you’ve got a satellite map but no underwater map, for instance.”
On the navigation side, Miller’s team picked up on work began by the MIT Marine Robotics Group, led by John Leonard, to develop diver-AUV teaming algorithms. With their navigation algorithms, Leonard’s group ran simulations under optimal conditions and performed field testing in calm waters using human-paddled kayaks as proxies for each divers and AUVs. Miller’s team then integrated these algorithms right into a mission-relevant AUV and commenced testing them under more realistic ocean conditions, initially with a support boat acting as a diver surrogate, after which with actual divers.
“We quickly learned that you just need more sensing capabilities on the diver if you think about ocean currents,” Miller explains. “With the algorithms demonstrated by MIT, the vehicle only needed to calculate the gap, or range, to the diver at regular intervals to resolve the optimization problem of estimating the positions of each the vehicle and diver over time. But with the actual ocean forces pushing all the things around, this optimization problem blows up quickly.”
On the perception side, Miller’s team has been developing an AI classifier that may process each optical and sonar data mid-mission and solicit human input for any objects classified with uncertainty.
“The thought is for the classifier to pass along some information — say, a bounding box around a picture — to the diver and indicate, “I believe this can be a tire, but I’m unsure. What do you think that?” Then, the diver can respond, “Yes, you have got it right, or no, look over here within the image to enhance your classification,” Miller says.
This feedback loop requires an underwater acoustic modem to support diver-AUV communication. State-of-the-art data rates in underwater acoustic communications would require tens of minutes to send an uncompressed image from the AUV to the diver. So, one aspect the team is investigating is learn how to compress information right into a minimum amount to be useful, working throughout the constraints of the low bandwidth and high latency of underwater communications and the low size, weight, and power of the business off-the-shelf (COTS) hardware they’re using. For his or her prototype system, the team procured mostly COTS sensors and built a sensor payload that will easily integrate into an AUV routinely employed by the U.S. Navy, with the goal of facilitating technology transition. Beyond sonar and optical sensors, the payload features an acoustic modem for ranging to the diver and several other data processing and compute boards.
Miller’s team has tested the sensor-equipped AUV and algorithms around coastal Recent England — including within the open ocean near Portsmouth, Recent Hampshire, with the University of Recent Hampshire’s (UNH) Gulf Surveyor and Gulf Challenger coastal research vessels as diver surrogates, and on the Boston-area Charles River, with an MIT Sailing Pavilion skiff because the surrogate.
“The UNH boats are well-equipped and might access realistic ocean conditions. But pretending to be a diver with a big boat is difficult. With the skiff, we will move more slowly and get the relative motion in tune with how a diver and AUV would navigate together.”
Last summer, the team began testing equipment with human divers at Michigan Technological University’s Great Lakes Research Center. Although the divers lacked an interface to feed back information to the AUV, each swam holding the team’s tube-shaped prototype tablet, dubbed a “tube-let.” The tube-let was equipped with a pressure and depth sensor, inertial measurement unit (to trace relative motion), and ranging modem — all needed components for the navigation algorithms to resolve the optimization problem.
“A challenge during testing was coordinating the motion of the diver and vehicle, because they do not yet collaborate,” Miller says. “Once the divers go underwater, there isn’t a communication with the team on the surface. So, you’ve got to plan where to place the diver and vehicle so that they don’t collide.”
The team also worked on the perception problem. The water clarity of the Great Lakes at the moment of 12 months allowed for underwater imaging with an optical sensor. Caroline Keenan, a Lincoln Scholars Program PhD student jointly working within the laboratory’s Advanced Undersea Systems and Technology Group and Leonard’s research group at MIT, took the chance to advance her work on knowledge transfer from optical sensors to sonar sensors. She is exploring whether optical classifiers can train sonar classifiers to acknowledge objects for which sonar data doesn’t exist. The motivation is to cut back the human operator load related to labeling sonar data and training sonar classifiers.
With the internally funded research program coming to an end, Miller’s team is now looking for external sponsorship to refine and transition the technology to military or business partners.
“The fashionable world runs on undersea telecommunication and power cables, that are vulnerable to attack by disruptive actors. The undersea domain is becoming increasingly contested as more nations develop and advance the capabilities of autonomous maritime systems. Maintaining global economic security and U.S. strategic advantage within the undersea domain would require leveraging and mixing the very best of AI and human capabilities,” Miller says.