AI-tainted welfare report forces partial refund: what every agency should learn
Deloitte agreed to repay the final installment of an AU$440,000 contract after its report for Australia's Department of Employment and Workplace Relations (DEWR) was found to include fake citations, phantom footnotes, and a fabricated quote from a Federal Court judgment.
The report covered the Targeted Compliance Framework, the IT-driven system that penalizes welfare recipients who miss obligations such as job search appointments. After publication, multiple fabrications were identified, triggering a correction and a public disclosure of AI use.
What happened
DEWR re-uploaded a corrected version of the report, removing more than a dozen bogus references, fixing typos, and revising text. The updated methodology now states Deloitte used a "generative AI large language model (Azure OpenAI GPT-4o) based tool chain" hosted in DEWR's Azure tenancy to fill "traceability and documentation gaps."
The original version cited imaginary works by real academics and included a fictional extract from the robo-debt case Amato v Commonwealth, even misnaming "Justice Davis" and quoting nonexistent paragraphs. DEWR maintains the substance and recommendations are unchanged. Deloitte's internal review reportedly attributed the gaffes to human error, though the volume of hallucinated references raised broader questions about process control.
Key point: AI assisted core analysis and cross-referencing, but that use wasn't disclosed up front. For government, that's a governance issue, not a typo.
Why this matters for government buyers
Trust is the product. If citations and legal quotes can't be verified, recommendations lose weight and decisions become exposed to audit and legal challenge. Undisclosed AI use also creates procurement and privacy risks if models, data handling, and hosting aren't specified in the contract.
This isn't an argument against AI. It's a case for disciplined disclosure, verification, and accountability when vendors use it on taxpayer-funded work.
What to require in future RFPs and contracts
- AI disclosure clause: Name the model(s) and version(s), hosting environment, data flows, and any third-party tools. Include a change-control process for model updates.
- Verification standard: No AI-synthesized citations. All references must resolve to real sources with working links, DOIs, or official citations. Vendors supply a "sources pack" (PDFs/screenshots of quoted pages).
- Reproducibility: Require a locked archive of prompts, outputs, and decision logs for anything AI-assisted, retained for audit.
- Human accountability: Name a lead author and quality reviewer who sign off that every citation and legal quote was checked against the primary source.
- Quality gates: Separate checks for facts, references, and law. Quotes from judgments must be validated against the official record.
- Data protection: AI must run in the agency's approved tenancy with no data leaving the jurisdiction. No training on agency data without explicit permission.
- Use-case constraints: For legal and policy analysis, restrict models to retrieval from agency-approved repositories. No freeform generation of legal citations.
- Audit rights and remedies: Withhold final payment until QA evidence is provided. Include clawbacks, mandatory rework at vendor cost, and professional indemnity coverage for errors.
Operational checklist for AI-assisted reports inside government
- Publish an AI use note with each report stating what was AI-assisted and how it was checked.
- Keep a live bibliography with DOIs or official links; store local copies of cited pages.
- Run an automated citation scan and a human spot-check before release.
- For legal material, verify against official sources such as AustLII.
- If using Azure OpenAI, document model identity and update cadence; see Microsoft's model guidance for context: Azure OpenAI models.
What this means for agency leaders
Treat AI like any other high-risk tool: define scope, disclose use, and verify outputs. If a vendor uses AI to "fill gaps," they must prove every filled gap traces back to a real source.
Set the bar now. Make disclosure and verification a condition of payment. You'll protect program decisions, safeguard public trust, and avoid weekend re-uploads.
Upskilling your team
If your staff needs practical training on AI policy, auditing, and prompt-related risk, explore role-specific options here: Complete AI Training - Courses by job.
