Stanford Scholars Train Generative AI To Be Better Creative Collaborators
Date: March 10, 2026
The AI art debate keeps bouncing between spammy outputs and full automation. Most creatives want something else: a model that listens, responds, and builds with you. The hurdle is control. Ask for a "house," you get one. Ask for a red house with four front windows, a chimney, and ivy on the left side, and the model drifts.
Stanford researchers in computer science, psychology, and education are fixing that drift by building a shared "conceptual grounding" between humans and models-so you can steer with precision and get production-quality visuals: illustrations, diagrams, even animations.
"While the models seem amazing, they are terrible collaborators," says Maneesh Agrawala, professor of computer science at Stanford. "Creators have no way of knowing what the AI will produce when given a certain text prompt. If you ask for a suburban single-family home, it generates a modern duplex."
Authoring original work is a chain of decisions. Without a shared set of concepts, nuance gets lost. The team's goal is simple: make your intent legible to the model, and make the model's logic legible to you.
How they're tackling it
The researchers are approaching the problem from two angles:
- Decoding human collaboration: running studies of people co-creating visuals, then analyzing chat logs and sketches to see how shared concepts emerge in the process.
- Building open-source tools: turning those insights into systems that let you control layout, pose, scene logic, and edits-without guesswork.
"If we want to build AI systems that understand how humans think during creative projects, we should start by learning as much as we can from the way that people establish common conceptual ground with each other," says Judith Fan, assistant professor of psychology at Stanford.
Tool 1: ControlNet for layout and pose
ControlNet teaches text-to-image diffusion models about spatial composition. It mirrors how artists work: first block in the scene, then add detail. Today's models often fumble pose and arrangement; ControlNet gives you handles to fix that. Source code and examples live on the project's repository: ControlNet on GitHub.
- Start with a rough: feed a sketch, depth map, or pose as the "blocking" pass to lock composition.
- Then refine: add the "detailing" pass for materials, lighting, and style without breaking layout.
- State constraints clearly: color, part counts, side-specific details (e.g., "ivy on the left façade only").
- Iterate in small moves: adjust one variable at a time to keep changes predictable.
Tool 2: FramePack for multi-scene 3D video
FramePack generates 3D videos from a text prompt by prioritizing scenes by their importance to the story-like a human editor allocating time to key beats. This helps keep attention where it matters instead of spreading effort thin across filler shots.
- Write a beat sheet: list scenes, goals, and the emotional or narrative "weight" of each.
- Tell the model what to emphasize: "Scene 3 is the reveal-allocate the most frames and camera movement here."
- Specify continuity: characters, props, and lighting that must persist across scenes.
Tool 3: Neuro-symbolic scene coding for transparency
The team is also building a neuro-symbolic pipeline: models map your natural language into a visual scene coding language, execute it, and render a 3D scene. You can inspect or edit the code and prompt the AI to update the program at any time. This adds reasoning and clarity to a process that usually feels like a black box. If you want the broader concept, see IBM Research on neuro-symbolic AI.
- Ask for code in the loop: object lists, transforms, materials, and constraints surfaced as editable parameters.
- Keep a glossary: agreed names for assets, colors, and styles to reduce ambiguity across iterations.
- Lock what matters: freeze dimensions or camera rigs so style changes don't break structure.
Why this matters for creatives
This shared grounding opens practical paths across design, simulation, animation, robotics, and education. The team is already collaborating with Roblox on safe text-to-3D object generation that respects game rules (e.g., preventing weapon creation in nonviolent experiences). The bigger picture: fewer surprises, faster alignment, and a tighter loop between your intent and the final render.
Prompts and patterns you can use today
- Constraint-first (static image): "Red two-story house, four front-facing windows symmetrically arranged, brick chimney right side, ivy covering the left façade only, centered in frame, 35mm perspective, overcast lighting."
- Rough-to-detail (ControlNet): "Use this pose/depth map to lock composition. Keep window count and positions fixed. Apply [style] detailing with warm dusk lighting-do not alter layout."
- Story-first (video): "Scenes: 1) Establish (low weight), 2) Conflict (medium), 3) Reveal (high-longer duration, close-ups, dolly-in), 4) Resolve (medium). Maintain character outfit A and prop B across all scenes."
- Code-in-the-loop (3D scene): "Generate scene program and list all parameters. Pause for edits before render. Variables to expose: positions, scales, materials, camera path, and light intensity."
Keep learning
If you want more practical workflows and case studies, browse our curated hub: AI for Creatives. You can also watch the research team's recent discussion from the Hoffman Yee Symposium at Stanford HAI to hear the latest findings.
Your membership also unlocks:
