NVIDIA engineers tested whether a general-purpose AI coding agent could build a real-time edge application using the same tools and documentation available to a human developer. The experiment, detailed in a new technical post, produced a working endoscopic tool segmentation app for NVIDIA Holoscan in about 40 minutes of agent processing time - and the team published the workflow so other developers can replicate it.
Holoscan is NVIDIA's platform for real-time AI at the edge, used in fields like medical imaging and robotics. Its companion repository, HoloHub, contains reference applications and reusable components. The team's goal was to see if a coding agent could navigate that ecosystem independently, using the Holoscan CLI, HoloHub documentation, and specialized development skills.
An agent-driven development workflow
The workflow runs in iterations. The engineer defines a goal and constraints, the coding agent inspects relevant examples and implements code, and the engineer reviews the results before setting the next objective. The team used Codex with GPT-5.6 sol max mode for the experiment.
The first prompt asked the agent to create a new Python application that reused an existing MONAI endoscopic tool segmentation model and sample video, with a HoloViz overlay showing segmentation masks and statistics. The agent read the app lifecycle skill, examined similar HoloHub examples, and generated the application scaffold through the CLI. It built and ran the app end to end in 40 minutes.
A second iteration added a benchmark mode that records latency and plots results, giving the developer repeatable performance measurements. The third iteration focused on reducing latency, and the agent cut mean application-path latency by 33.6% and raised rendered throughput from 204 to 306.9 FPS by optimizing dashboard overhead rather than skipping inference frames.
What the ablation study shows
The team also ran the same first-iteration prompt under three different resource configurations. With the full stack - CLI, skills, and documentation - the agent took 40 minutes and used 11M tokens. Without skills, processing time rose to 65 minutes and token use to 20M, as the agent resorted to generic Bash tools and more trial-and-error. Without both CLI guidance and skills, the agent still found the CLI from codebase examples but produced lower-quality code: it embedded model configs incorrectly, created a Dockerfile that ignored the existing HoloHub base image, and used custom PyTorch inference instead of optimized Holoscan operators, making the result 2.6x slower.
The development iterations show what the workflow produces. The agent and the developer use the same CLI commands, so the engineer can inspect every step. The final application runs end to end, offers visual, smoke, and benchmark modes, preserves model weights, and records reproducible benchmark evidence.
Why this matters for IT and Development
The practical takeaway is that the quality of AI-assisted development depends heavily on the scaffolding around the agent. A well-structured CLI, project-specific skills, and clear documentation reduced token usage by nearly half and cut processing time by roughly 40% compared to an unguided agent. For teams evaluating Generative Code tools, this suggests the bottleneck isn't just model capability - it's how much context and structure you provide. The same pattern applies to AI for IT & Development workflows: agents perform best when the environment encodes the right operations and the engineer reviews each iteration. The HoloHub skills and CLI are available on GitHub, and the team has published the full workflow for developers who want to try it.
Your membership also unlocks:
