AI-Fueled Scandal: Deloitte Repays Over €250,000 After Fake References Surface in Government Report
A major consulting engagement on Australia's welfare penalty automation ran into a basic failure: fake references created by generative AI made it into the final report. Deloitte has repaid an amount exceeding €250,000 (about AUD 440,000) after the issues were flagged.
The Department of Employment and Workplace Relations commissioned the work. After publication, academics spotted citations to articles that didn't exist, including works attributed to a real professor that were never written.
What actually happened
Large sections of the original report were drafted with a generative AI "toolchain" (Azure OpenAI GPT-4o). The firm did not properly verify sources before delivery.
Following scrutiny, Deloitte and the ministry issued an updated 273-page report, removing fabricated references and correcting footnotes. Deloitte said it would reimburse for the "final version" of the contract, while the ministry indicated the audit's recommendations remain in play.
Why this matters for public sector leaders
Procurement trusts vendors to get the basics right: cite real sources, verify facts, and document methods. AI can speed up drafting, but it can also invent details and citations if left unchecked.
The fix is not to ban AI; it's to raise the bar on verification, disclosure, and accountability. Below is a fast, practical toolkit you can apply to any AI-assisted deliverable.
Procurement and delivery safeguards to put in contracts now
- AI-use disclosure: Vendors must declare if AI is used, where, and why. Include model names, versions, and hosting (e.g., Azure OpenAI GPT-4o).
- Source-of-truth rule: Every claim tied to a verifiable primary source or dataset. No unsourced assertions.
- Citation validation: Provide working links or DOIs. Broken or non-existent citations fail acceptance.
- Human verification: Named reviewers sign off that facts and references were checked by humans.
- Prompt and method logs: Require secure logs of prompts, settings, and retrieval sources for audit.
- Independent fact-check: Mandate a second-party or internal QA pass with a documented checklist.
- Reproducibility: If AI assisted analysis, vendor must reproduce results on request.
- Holdbacks and remedies: Tie final payment to passed checks; include clawbacks for fabricated citations.
- Audit rights: Reserve the right to review underlying notes, tools, and data lineage.
Technical controls your teams should require
- Reference checker: Automated validation of citations, URLs, DOIs, and publication metadata.
- Retrieval with citations: If AI is used, require retrieval-augmented generation with source links surfaced in-line.
- Model policy: Approved model list and version control; document updates that could alter outputs.
- Usage logs: Keep model call logs (timestamps, versions, temperature) to explain discrepancies.
- Quality thresholds: Define acceptable "hallucination" rates in pilots; zero tolerance for fabricated references in final deliverables.
- Peer review: A human subject-matter expert reviews any AI-assisted sections before submission.
Governance frameworks to lean on
Anchor your policy to an established standard to avoid starting from scratch and to align with auditors and oversight bodies.
- NIST AI Risk Management Framework for risk controls, measurement, and governance patterns.
Rapid checklist for reviewing AI-assisted reports
- Spot-check 10-20% of citations; confirm each source exists and supports the claim.
- Click every link in the executive summary and conclusion.
- Search for quoted or paraphrased passages and confirm proper attribution.
- Ask for the prompt/method log and the list of models and versions used.
- Recreate two key findings using the provided methods and data.
- Escalate any non-existent or mismatched citations; withhold acceptance until resolved.
What to tell vendors this week
- Send an addendum: disclose AI use by section, tool, and version, or confirm none was used.
- Deliver a citation dossier: live links, DOIs, and PDFs where allowable.
- Provide a QA attestation signed by a named reviewer.
- Agree to holdbacks tied to passed citation and fact checks.
- Commit to immediate corrections and cost coverage for fabricated references.
Bottom line
AI can draft text; it cannot carry your due diligence. Contracts must demand proof of sources, logs of methods, and human sign-off. Tie payment to evidence, and this problem goes away.
If your department needs practical training on AI procurement, verification, and QA workflows, see our programs here: Complete AI Training - Courses by Job.
Your membership also unlocks:
