U.S. Air Force cadet Joshua Lynch, with no prior coding experience, built a functional software application using only AI chatbots over a three-month period. The project, part of the Department of the Air Force-MIT AI Accelerator's Phantom Program, tested whether nontechnical service members can use vibe-coding - relying entirely on prompts to guide generative AI - to solve military problems without the traditional software development pipeline. The results highlight both the promise and the sharp limits of AI-assisted development for non-experts, a finding with direct implications for IT and development teams that may soon see more prototype work coming from business-side colleagues.
The effort, documented by MIT Lincoln Laboratory researcher Laura Niss, aimed to see if someone familiar with a military problem space could bypass time and cost constraints by generating code through chatbots like Anthropic's Claude, OpenAI's ChatGPT, and Google's Gemini. Lynch set out to build an application he called the Remote Operating Modular Augmentation Device (ROMAD-AI), originally envisioned to assist with target recognition, surveillance, and communication on the battlefield. As he gained experience, he learned that generative code creation required breaking problems into small pieces, framing questions clearly, and steering conversations back on track when chatbots lost focus.
Lynch, who had no coding background, worked with the paid models of all three chatbots through their web browser chat functions, later using Google AI Studio App for the final build. Over time, he scoped down the project from a battlefield assistant to a document-processing tool that could analyze tactical maps and generate mission-planning documents by interfacing with a vision-language model. "I was quite impressed with this final product, and it showed me how powerful these systems can be at prototyping designs from nonexperts," Niss said.
Lessons from the vibe-coding experiment
The project timeline was dominated by learning to recognize and work around the chatbots' limitations. Lynch found that AI models often lacked hierarchical focus, modifying unrelated code sections. He also noted inaccuracies when the chatbots discussed topics he knew well, making them better as tutors than as authoritative sources. Niss observed a shift in Lynch's perception of AI over the three months - his ambitious initial goal gave way to a much more restricted scope as he understood the technology's current capabilities.
The final prototype did not include all the features Lynch originally intended, and in its current form was not secure enough for battlefield use. But it proved that a nontechnical user could produce a viable application. "For me, this project reinforced the expanse between experts in different fields," Niss said. "No matter how good AI gets, I think we'll always need to collaborate to get to the best solutions for the most important problems."
Security and code review remain bottlenecks
One security misstep underscored the risks of AI-generated code. Lynch did not realize that the final application was sending input documents to a Gemini AI model for analysis rather than parsing them locally. Improper vetting of code can introduce significant security vulnerabilities, and the project highlighted that while AI can produce large amounts of functional code, code review remains a critical bottleneck. For IT and development teams, this means that prototypes from non-expert colleagues will still require rigorous inspection before they can be trusted in production environments.
Why this matters for IT and development professionals
The experiment shows that generative AI tools can lower the barrier for domain experts to create working software prototypes, potentially increasing the volume of ideas that reach technical teams. However, the gap between a chatbot-generated prototype and a secure, production-ready application is wide. IT and development professionals will need to establish clear review processes for code generated by nontechnical users, especially when sensitive data or critical systems are involved. The project also suggests that AI-assisted coding is most effective when used as a communication tool - helping non-experts convey problems and possible solutions to technical experts, rather than replacing the need for skilled developers.
Your membership also unlocks:
