MIT ultrasound wristband revolutionizes AR, VR and robotic movement

Your hand turns daily life into a quiet feat of engineering. Each time you scroll, pinch, point or grab, 34 muscles, 27 joints and more than 100 tendons and ligaments work together. That silent teamwork lets you tie shoes, hold a pencil, play piano keys and send a text without thinking much about it.

For robots and virtual reality systems, that same ease has been hard to copy. Cameras can lose sight of your fingers. Sensor gloves can feel stiff. Muscle-signal devices can miss tiny shifts in motion. Now, MIT engineers have built an ultrasound wristband that can track the hand from the inside, using the motion of muscles and tendons in the wrist.

The device uses ultrasound images and artificial intelligence to predict the position of all five fingers and the palm in real time. In demonstrations, a wearer used the band to control a robotic hand wirelessly. When the person pointed, pinched or tapped, the robot followed.

The wristband works by watching the body’s own control cables. When you move your fingers, tendons and muscles in the wrist shift in clear patterns. The device images those changes with ultrasound, much like medical ultrasound looks inside the body.

“The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers,” said Gengxi Lu. “So the idea is, each time you take a picture of the state of the strings, you’ll know the state of the hand.”

The system includes an ultrasound sticker about the size of a smartwatch. It pairs with small electronics and a soft hydrogel layer that helps the sensor stay connected to the skin. The wristband records continuous images of the wrist as the wearer moves.

An AI model then translates those images into hand positions. The system tracks 22 degrees of freedom, meaning it can follow many angles and motions across the fingers and palm. It updates fast enough to make virtual or robotic movements feel smooth.

Hand tracking already exists, but each method has limits. Camera systems need a clear view. They can struggle when your hand passes behind an object or leaves the camera’s field of view. Lighting and angle can also cause problems.

Sensor gloves can capture movement more directly, but they may interfere with natural touch. If a glove covers your fingers, you may not feel objects the same way. That matters for tasks that depend on comfort, grip and delicate motion.

Other systems use electrical signals from forearm muscles. These can detect broad gestures, but they often struggle with subtle movement. They may know that a finger is pinching, but not the exact path between open and closed.

MIT’s approach avoids many of these issues. It does not need to see the hand directly. It also leaves the fingers bare, so the wearer can still feel and touch normally.

To teach the system, researchers matched ultrasound images with hand positions. Volunteers wore the wristband while cameras recorded their hand movements from different angles. The team then labeled parts of the ultrasound images that matched specific finger and palm motions.

This training helped the AI learn the hidden link between wrist structures and hand motion. Once trained, the model could look at new ultrasound images and predict the wearer’s gesture.

In the study, eight volunteers with different hand and wrist sizes tested the device. They made many gestures and grasps, including the signs for all 26 letters in American Sign Language. They also held objects such as a tennis ball, plastic bottle, scissors and pencil.

The wristband tracked hand position with strong accuracy. It followed not just fixed poses, but smooth transitions between gestures. That continuous tracking is important for realistic robot control and virtual interaction.

In one demonstration, a person wearing the band controlled a robotic hand. As the wearer moved their fingers, the robot copied the motion in real time. The interaction looked like a wireless marionette, but without strings.

The robot played a simple tune on a piano by following the wearer’s finger taps. It also shot a small basketball into a desktop hoop. These tasks sound playful, but they show something serious: the device can turn natural hand movement into useful machine control.

The wristband also worked in virtual settings. A wearer could pinch to enlarge or shrink an object on a screen. They could also rotate or move virtual objects using natural gestures.

“We think this work has immediate impact in potentially replacing hand tracking techniques with wearable ultrasound bands in virtual and augmented reality,” said Xuanhe Zhao. “It could also provide huge amounts of training data for dexterous humanoid robots.”

One of the biggest future uses may involve training robots. Humanoid robots need hand skills to work in homes, hospitals, factories and labs. But teaching them fine motor skills takes enormous amounts of data.

The MIT team hopes the wristband can collect hand motion data from many users. People with different hand sizes, finger shapes and movement styles could help build a large motion library. Robots could then learn from that human movement.

That matters for tasks where small differences count. A robot that assists in surgery, for example, must move with care and precision. A robot helping at home must grasp fragile objects without crushing them.

The wristband may also help people interact more naturally with machines. Instead of pressing buttons or using joysticks, users could control tools through ordinary hand motions.

The current system still has room to improve. Zhao’s team plans to shrink the hardware further and train the AI on more users. A future version could become lighter, longer lasting and easier to wear.

The researchers also want the device to work across more hand types without long personal training sessions. Larger datasets may help the AI generalize from one person to another.

If those goals succeed, ultrasound wristbands could become a new kind of interface. They could help with virtual reality design, gaming, remote robotics, medical training and assistive technology.

“We believe this is the most advanced way to track dexterous hand motion, through wearable imaging of the wrist,” Zhao said. “We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands.”

This research could change how people control robots and virtual environments. A wearable ultrasound band could replace camera-based tracking in places where cameras fail. It could also remove the need for bulky gloves that limit touch and comfort.

In robotics, the technology could create huge datasets of natural hand motion. Those datasets may help humanoid robots learn delicate tasks, including object handling and surgical movements. The device could also support prosthetics and assistive systems that need precise hand-intent tracking.

For virtual reality and augmented reality, the wristband could make digital interaction feel more natural. Users could grasp, pinch and move objects without external cameras. That may improve games, design tools, training systems and remote collaboration.

Research findings are available online in the journal Nature Electronics.

The original story "MIT ultrasound wristband revolutionizes AR, VR and robotic movement" is published in The Brighter Side of News.

Like these kind of feel good stories? Get The Brighter Side of News' newsletter.