AI opens 3D modeling to blind and low-vision coders
3D modeling is essential to design, engineering, and fabrication. It's also been one of the hardest tasks for blind and low-vision programmers to do without help. A new system called A11yShape changes that by turning visual steps into code, structure, and precise language.
Built on OpenSCAD and using GPT-4o for contextual assistance, A11yShape lets users create, modify, and verify models independently. The system was developed through participatory design with input from blind researchers and was presented at the ACM SIGACCESS ASSETS conference.
How A11yShape works
A11yShape links four synchronized views of the same object: source code, a semantic hierarchy, AI-generated descriptions, and rendered images. Select a part in one view, and it highlights across the others. For screen reader users, audio cues signal changes. For low-vision users, highlighted parts render semi-transparent to keep overlaps visible.
To keep language grounded, the AI receives the relevant code and multiple renders from six standard angles. It returns a concise overview, part-by-part details, and suggested code edits. This cuts guesswork and limits hallucinated components.
The interface in three panels
- Code Editor: Screen reader-friendly. Clear syntax errors. Built for incremental changes.
- Code Changes List: Summaries of edits by you or the AI (e.g., cylinder height/diameter adjustments), produced by a separate language model diffing code versions.
- AI Assistance Panel: Conversational guidance. Ask questions, request edits in plain language, and review versions with traceable suggestions.
The Model Panel adds a hierarchical list of components-think "landing gear" with legs and base. Users can inspect parts via keyboard, switch views, or ask the AI to adjust the camera.
Validation: accuracy and trust
A validation study with 15 sighted modelers rated AI descriptions of eight models across five metrics. Average scores ranged from 4.11 to 4.52 on a five-point scale. The highest marks went to avoiding hallucinations; clarity and spatial relationships were also strong (both above 4.2). Geometric precision and completeness were slightly lower but still above 4.1, suggesting reliable summaries with room to tighten numeric detail.
Blind and low-vision study: learning by building
Four blind or low-vision programmers (ages 21-32) with prior coding experience but no OpenSCAD background used A11yShape over ~7.5 hours each across three sessions. They moved from syntax basics, to guided tasks, to self-directed projects. Outputs included a bacteriophage analysis, a Tanghulu skewer, a robot, a circuit board, a rocket, a cart, and a helicopter-12 models in total.
One low-vision participant, Alex, built a helicopter by shifting to an incremental workflow: assemble a part, verify with descriptions and cross-highlighting, then adjust placement. The final model had clear structure-body, landing gear, rotors-while some alignment issues remained. Fine spatial tuning without visual or tactile feedback is still tough.
Usability and friction points
On the System Usability Scale, A11yShape averaged 80.6 (high). Participants called the experience empowering and credited the editor-AI pairing for making the work feasible. A lower outlier score reflected learning OpenSCAD, not the interface itself.
Challenges centered on cognitive load. Long descriptions can overwhelm, and holding multiple model versions in working memory is taxing. Spatial reasoning-estimating coordinates, tuning proportions, and tracking intersections-demands careful iteration. Participants adapted with two patterns: write most code and use AI for verification, or start with AI-generated structure and refine manually. Frequent, small edits won out.
Why this matters for research and engineering teams
- Demonstrates that code-first 3D modeling with language support can reduce reliance on sighted assistance.
- Cross-representation highlighting (code ⇄ hierarchy ⇄ text ⇄ renders) grounds AI output and improves trust.
- Structured camera views and targeted descriptions reduce hallucinations and ambiguity.
- Incremental workflows with versioned summaries help manage cognitive load.
Limitations and next steps
- Numeric precision and completeness can improve, especially for tight tolerances and part intersections.
- Spatial fine-tuning is still hard without visual or tactile feedback; haptics and 3D-printed checkpoints may help.
- Opportunities: integrate 3D printing pipelines, circuit prototyping, and fabrication feedback loops.
- Extend cross-representation linking to other domains: data visualization, slide layout, web structure.
Who's behind it
The work involves researchers from the University of Texas at Dallas, the University of Washington, Purdue University, Stanford University, the University of Michigan, MIT, The Hong Kong University of Science and Technology, and Nvidia. A blind PhD student at MIT, Gene S-H Kim, contributed user insights that shaped the system's design. Findings were presented at the ACM SIGACCESS ASSETS conference.
Practical takeaways you can apply right now
- When building AI assistance for spatial tasks, pair language with synchronized code and structured views.
- Ground descriptions with fixed camera sets and the exact code segment under discussion.
- Provide change logs driven by code diffs so users can verify intent vs. outcome quickly.
- Favor short, layered descriptions with drill-down details to manage cognitive load.
Learn more
Explore the underlying modeling approach at OpenSCAD, and watch for peer-reviewed updates via the ACM SIGACCESS ASSETS venue. If you're building LLM-driven tooling and want to sharpen your prompts and evaluation habits, see this prompt engineering roundup: Complete AI Training - Prompt Engineering.
Your membership also unlocks:
