In today’s world, artificial intelligence chatbots resembling ChatGPT and Claude can perform many functions, resembling composing work emails and planning travel itineraries. These chatbots are systems built around large vision-language models (VLMs): AI trained on a large dataset that features books, web sites, code, and pictures.
The AI algorithms are then refined on massive amounts of human-generated feedback to follow instructions and avoid harmful or unwanted output, and use that “knowledge” to provide text or images based on input from a user. Although chatbots have clear limitations, they might be very helpful for a wide selection of tasks, including in some areas that traditionally require specialized skills, like computer programming.
As a part of a project for the U.S. Department of the Air Force–MIT AI Accelerator‘s Phantom Program, U.S. Air Force cadet Joshua Lynch — with the assistance of his mentor, Laura Niss, a technical staff member within the Embedded and AI Systems Group at MIT Lincoln Laboratory — wanted to find out if, as an entire novice to coding, he could develop a totally functional program. He used a process called “vibe-coding,” during which a user relies entirely on prompts to guide a generative AI chatbot to jot down and refine code.
His motivation was to empower anyone acquainted with the military problem space, no matter their technical background, to advance their ideas for useful software applications, essentially bypassing the time and price constraints of the normal military software development pipeline. Lynch aimed to construct his own application while Niss monitored his experience with the technology.
“The Phantom student desired to see if he could create a useful application through self-identified vibe-coding, with none previous experience,” Niss says. “Inside this project, I wanted to grasp how his perception of AI modified over time with use. We each wanted to grasp higher where and the way AI could possibly be utilized by nontechnical users within the military.”
Lynch got down to see if, starting with no coding skills and using chatbots, he could create an application specific to his sort of tactical team to assist reduce collateral damage while enhancing survivability within the broader mission. This application would offer capabilities including AI-assisted goal recognition; modular intelligence, surveillance, and reconnaissance; autonomous striking; and communication management on the battlefield.
Throughout the project, Lynch accomplished several skilled development courses in AI and familiarized himself with each military and nonmilitary uses of the technology. For the idea for his code generation, he used the paid models of three AI chatbots: Anthropic’s Claude, OpenAI’s ChatGPT, and Google’s Gemini. Most of this work was done only through the chatbots’ principal chat function on an internet browser, not as an integrated system inside a development environment, as is standard now. The ultimate application was produced using Google AI Studio App, which might create applications that interface with the Gemini application programming interface and has AI integrated in the event environment.
Over three months, Lynch worked with these models to construct his application, called the Distant Operating Modular Augmentation Device (ROMAD-AI). During this time, he learned several methods to enhance the code output. For instance, he often encountered difficulties with the AI chatbots lacking hierarchical focus and modifying unrelated code sections. He discovered it was essential to interrupt problems into small parts, frame questions clearly, and steer conversations back on topic after they stray too removed from the target.
Learning to acknowledge the chatbots’ limitations and effectively work around them took up a lot of the project timeline. As Lynch gained more experience with the chatbots, limitations within the AI capabilities and time for development caused him to re-scope the project, moving it from an application that would assist on the battlefield to at least one that would perform basic document processing, resembling analyzing tactical maps of battlefields and generating mission-planning documents through an interface with a VLM-powered chatbot. While the resulting prototype didn’t perform all capabilities Lynch originally set out to incorporate (and in its current iteration was not secure for the specified use case), it proved the aptitude and usefulness of such an application for service members.
“I used to be quite impressed with this final product, and it showed me how powerful these systems might be at prototyping designs from nonexperts,” Niss says. “I’m now of the opinion that these might be powerful tools for nontechnical experts to convey problems and possible solutions to technical experts, and aid in communicating desired outcomes.”
Niss observed the change in Lynch’s perspective of AI language models during his experience. After starting with a formidable goal, Lynch gained understanding of the capabilities of current technology and significantly scoped down his expectations by the top of the project period. Measures of his perceptions of different AI systems over time and across system updates were particularly interesting to Lynch and Niss, with Claude showing more stability than ChatGPT across traits resembling likeability, anthropomorphism, and perceived intelligence. Lynch found AI to be a helpful tutor, but noted its inaccuracies on topics he knew well.
The project showed that AI chatbots can empower nontechnical service members to provide viable software applications for his or her unique problems, even though it works higher as a prototyping assistant than as a full production tool when handling sensitive information and for critical applications. Improper vetting of code may result in security risks, as demonstrated by an instance where Lynch didn’t realize that the ultimate application was sending the input documents to a Gemini AI model to research, relatively than parsing the documents locally on his computer. Although AI can generate significant amounts of functional code, code review stays a bottleneck on this space.
“For me, this project reinforced the expanse between experts in numerous fields,” Niss says. “Regardless of how good AI gets, I believe we’ll at all times must collaborate to get to the very best solutions for an important problems.”
Research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was completed under Cooperative Agreement Number FA8750-19-2-1000.