OpenAI's Pentagon Deal: What It Enables-and What It Risks
OpenAI has agreed to provide its systems for classified military use. That puts its models in the room where wartime and intelligence choices get made. Outside the company's San Francisco office, a small chalk protest captured the fear plainly: "Stand for liberty." "Please no legal mass surveillance."
A leaked internal memo reportedly said leadership will push "red lines" against mass domestic surveillance and autonomous lethal weapons. That sounds good. It means little unless those lines are converted into contract terms, technical controls, and independent oversight.
The core legal question: Who controls use?
Corporate policies are not enforceable. Government contracts are. If this work proceeds under FAR/DFARS or an OTA, the language in the SOW, clauses, and CDRLs will determine what the government can actually do-and what the vendor must refuse.
Because the work is classified, normal sunlight is gone. That raises the bar for precision: definitions, auditability, and termination rights must be unambiguous and testable.
Surveillance risk surface
Mass surveillance rarely starts as "mass." It starts as broad exceptions, unclear retention, and quiet interagency sharing. AI systems that summarize, correlate, or predict across datasets can move from targeted to sweeping fast.
Anchor any "red line" to existing law and oversight. For reference on foreign-intelligence collection that can touch Americans, see the Privacy and Civil Liberties Oversight Board's work on Section 702 of FISA.
Lethal autonomy risk surface
DoD policy requires human judgment in the use of force. But API wiring can drift into functionally autonomous behavior if guardrails are weak or brittle under operational stress.
Turn "red lines" into enforceable contract language
- Definitions that bite: "mass domestic surveillance," "target selection," "weapons release," "model training data," "operational data," and "derived data."
- Explicit prohibitions: no use for domestic bulk collection; no autonomous target identification, selection, or engagement; no facial recognition for law enforcement without explicit statutory authority.
- Purpose limitation: bind model use to enumerated missions and datasets; require written authorization for any new use case.
- Human-on-the-loop: mandate positive human control for any kinetic-relevant function, with documented decision checkpoints.
- Data boundaries: no ingestion of U.S.-person data absent lawful process; strict minimization; retention windows; deletion SLAs; no secondary training on operational data without approval.
- Audit and access: immutable logs; third-party audit rights; classified read-ins for vendor auditors; sampling rights over prompts, outputs, and system actions.
- Kill switch and suspension: immediate suspension triggers on policy violations, model drifts, or novel use discovery; clear path to termination for cause.
- Update control: require notice and approval for model/version changes; change-control board with government veto power.
- Liability and remedies: liquidated damages for barred uses; indemnity for unlawful surveillance claims; preservation obligations for investigations.
- Subcontracting and supply chain: flow-down of all constraints; disclosure and approval of all sub-processors.
Make the clauses real with technical controls
- Allow-listed endpoints within ATO'd enclaves; no external calls without explicit approval.
- Policy-enforced gating: classifiers that block prohibited tasks (e.g., target selection, identity correlation) before the model executes.
- Data loss prevention tuned for PII, biometrics, and geolocation; automatic redaction where lawful.
- Purpose-bound API keys and scoped tokens; per-mission secrets that expire.
- Immutable, append-only logging (hash-chained) for prompts, context, system actions, and outputs.
- Continuous red-teaming for surveillance and autonomy abuse cases; quarterly adversarial testing reports.
- Model cards and system cards specific to each classified integration, documenting known failure modes and mitigations.
Oversight that works under classification
- RACI chart that names the official who can say "no" and enforce it in production.
- Independent compliance officer with read-in, audit authority, and direct reporting to the mission commander and agency GC.
- Mandatory reporting windows: 24 hours for suspected prohibited use; 72 hours for confirmed incidents; 7 days for full RCA.
- External oversight: briefings to IG/PCLOB staff as permitted; minimum public transparency reports with aggregated metrics.
For government buyers and counsel
- Prefer contract structures that preserve data rights and audit (DFARS over loose OTAs when feasible).
- Mandate model version pinning; no silent upgrades. Every change gets a risk assessment and sign-off.
- Require holdback environments and test harnesses to validate "red line" enforcement before deployment.
- Tighten FOIA-sensitive disclosures without hiding material safeguards. Security through clarity, not obscurity.
- Map authorities: what statute, order, or policy authorizes each use? If the answer is fuzzy, the use is off-limits.
For company leadership and legal
- Make internal policies match the contract. If there's a gap, the contract wins-fix the policy.
- Stand up an internal refusal protocol with escalation paths that survive program pressure.
- Log every request that brushes a red line. Patterns matter in audits and hearings.
- Publish a public-facing commitment that mirrors the contract prohibitions. Stake your reputation to enforcement.
What to watch next
- Whether "red lines" appear as binding clauses with measurement and penalties-or stay as talking points.
- How the Pentagon responds to vendors that condition access on limits around surveillance and autonomy.
- Whether classified implementations ship with real kill switches, immutable logs, and third-party audits.
Resources
Bottom line: if OpenAI's tools enter classified workflows, the only guardrails that count are the ones you can read in the contract, test in staging, and enforce in production. Write them down. Wire them in. Audit them often.
Your membership also unlocks:
