Scientists create new AI system to help visually impaired coders

Three-dimensional modeling sits at the heart of modern design, engineering, and fabrication. Yet for blind and low-vision programmers, it has long remained one of the most difficult creative tasks to access independently. Unlike photo editing or coding, which can now be handled with screen readers and large language models, 3D modeling depends heavily on visual feedback. Every rotation, scale change, or placement decision usually requires seeing how shapes relate in space.

That gap is now beginning to narrow. A new AI-assisted system called A11yShape offers blind and low-vision programmers a way to create, inspect, and verify 3D models on their own. Developed through participatory design, including a blind co-author, the tool replaces visual dependency with code, structure, and carefully targeted AI descriptions. Instead of relying on a sighted helper to interpret a digital scene, you can explore a model through language, logic, and spatial reasoning supported by artificial intelligence.

A11yShape builds on OpenSCAD, a popular open-source tool that generates 3D models from code rather than direct manipulation. The system adds an accessible web interface and layers of AI assistance powered by GPT-4o. Its core strength lies in linking four representations of the same object: the source code, a semantic hierarchy of parts, AI-generated descriptions, and rendered images.

When you select a part in one view, that same component highlights across every other view. If you focus on a block of code, the system identifies the matching structure in the model and triggers a targeted description. For screen reader users, audio cues signal changes. For low-vision users, highlighted parts appear semi-transparent, allowing overlaps and connections to remain visible.

The interface includes three main panels. The Code Editor supports screen readers and logs syntax errors clearly. A Code Changes List summarizes edits made by you or the AI, such as changes to a cylinder’s height or diameter. These summaries come from a separate language model that compares earlier and updated versions of the code.

The AI Assistance Panel functions like a conversational guide. You can ask questions about the model, request edits in plain language, or review past versions. To answer accurately, the system sends the AI both the relevant code and multiple rendered views from six standard angles. The AI responds with a short overview, detailed descriptions of individual parts, and a list of proposed code edits.

To evaluate how reliable these descriptions were, researchers conducted a validation study with 15 sighted participants experienced in 3D modeling. Participants rated AI descriptions of eight models across five categories on a five-point scale. Average scores ranged from 4.11 to 4.52. The highest score measured avoidance of hallucinations, meaning the AI rarely described parts that were not present. Ratings for clarity and spatial relationships also exceeded 4.2. While geometric precision and completeness scored slightly lower, they still remained above 4.1, suggesting strong overall trustworthiness with room for refinement.

The Model Panel completes the system. It presents a hierarchical list of components grouped into meaningful units, such as landing gear made up of legs and a base. You can navigate this structure to inspect specific parts and switch between views using keyboard shortcuts. You can also ask the AI to adjust the camera angle when needed.

Researchers tested A11yShape with four blind or low-vision programmers in a multi-session remote study. All participants were men between 21 and 32 years old and had prior programming experience. None had used OpenSCAD or other 3D modeling tools before.

Over three sessions totaling about 7.5 hours per person, participants progressed from learning basic syntax to guided tasks and then free-form projects. Early exercises included inspecting a complex bacteriophage model using descriptions and semantic structure. Later, participants built objects such as a Tanghulu skewer and a robot. In the final session, they chose their own projects, including a circuit board, a rocket, a cart, and a helicopter.

Across the study, participants produced 12 distinct models. While some had minor flaws, the results showed that blind and low-vision users could independently create complex 3D objects when supported by layered feedback.

One participant, a low-vision programmer identified as Alex, built a helicopter. After an initial attempt generated an inaccurate shape, he shifted to an incremental approach. He constructed each part step by step, using AI descriptions and cross-representation highlighting to verify placement. By the end, the model included a recognizable body, landing gear, and rotors. Some alignment issues remained, showing the limits of spatial fine-tuning without visual or tactile feedback.

On the System Usability Scale, participants gave A11yShape an average score of 80.6, considered high usability. One participant noted that his lower score reflected the challenge of learning OpenSCAD rather than problems with the tool.

Participants described the experience as empowering. One said it showed that blind and low-vision users “can indeed create relatively simple structures.” Another called the system “revolutionary,” praising the editor and AI explanations.

"Participants also reported several recurring challenges. Long, dense descriptions created cognitive overload, especially during extended sessions. Some found it hard to hold multiple versions of a model in working memory while reading through pages of text. Understanding spatial relationships remained a major hurdle. Tasks like estimating coordinates, adjusting proportions, and tracking how parts intersected or overlapped demanded careful mental calculation," Dr. Liang He, assistant professor of computer science in the Erik Jonsson School of Engineering and Computer Science at the University of Texas at Dallas shared with The Brighter Side of News.

"However, participants adapted by developing different workflows. Some wrote most code themselves and used AI for verification. Others began with AI-generated structures and refined them manually. Incremental building and frequent verification became common strategies," Dr. He continued.

“This is a first step toward a goal of providing people with visual impairments equal access to creative tools, including 3D modeling,” said Dr. He.

Dr. He collaborated with researchers from the University of Washington, Purdue University, Stanford University, the University of Michigan, the Massachusetts Institute of Technology, The Hong Kong University of Science and Technology, and Nvidia Corp. The team presented its findings at ASSETS 2025, the international conference on accessible computing, held in Denver.

One of the co-authors, Gene S-H Kim, a blind PhD student at MIT, contributed user-centered insights that shaped the system’s design. “Every single time when he was working on his assignment, he had to ask someone to help him and verify the results,” Dr. He said, recalling a blind classmate’s struggles that inspired the work.

A11yShape demonstrates that accessible 3D modeling is achievable without sighted assistance. The approach could support education, workforce inclusion, and creative independence for blind and low-vision users.

Future development may link the tool to 3D printing, circuit prototyping, and fabrication pipelines, extending access from digital models to physical objects.

Researchers also see potential for applying cross-representation highlighting to other fields, including data visualization, slide design, and web layout.

Research findings are available online in the journal Association for Computing Machinery.

Like these kind of feel good stories? Get The Brighter Side of News' newsletter.