AI debacle is a wake-up call for corporate finance
Deloitte Australia faced public scrutiny after delivering a report to the Australian Department of Employment and Workplace Relations that included AI-generated errors. The original document reportedly contained a fabricated quote from a federal court judgment and references to nonexistent academic papers. A revised version has since been posted, and Deloitte agreed to refund part of the AU$440,000 (US$290,000) fee.
The department said "the substance" of the review remains, but that won't matter to finance leaders if controls fail. The message is simple: AI can speed up analysis, but it will also produce confident nonsense if you let it.
What happened, in brief
- Original report published in July; revised after errors were flagged by a university researcher.
- Errors included fake citations and an invented legal quote.
- Deloitte agreed to repay the final contract installment; the department kept the core findings.
Why this matters for finance teams
AI is useful, but it's not a source of truth. It mirrors your prompts and your controls. Without verification, it will fabricate citations, misstate facts, or overfit to biased inputs.
Generative models can "hallucinate," especially under vague prompts, weak retrieval, or poor data quality. For background on this failure mode, see OpenAI's guidance on reducing hallucinations here.
The risk isn't theoretical. Studies show many employees already make errors due to AI, often without clear approval or guardrails. If your team ships outputs with broken citations or inaccurate footnotes, your reputation and contract fees are on the line.
Minimum controls to implement now
- Human-in-the-loop review: Require domain experts to sign off on all AI-assisted deliverables. No exceptions for client-facing work.
- Citation verification: Automate link checks, DOI validation, and case-law lookups; reject any reference that can't be traced to an authoritative source.
- Prompt standards: Use approved prompt templates that demand sources, confidence levels, and uncertainty flags.
- Retrieval over recall: Pair models with a vetted knowledge base; block free-form generation on legal, regulatory, or policy topics without citations.
- Model and data inventory: Maintain a register of models, versions, training data provenance, and intended use cases.
- Adversarial testing: Red-team prompts to see how easily the model fabricates, leaks data, or accepts manipulated inputs.
- Usage policy and training: Clarify what's allowed, what's not, and where disclosure is required. Enforce with periodic audits.
- Logging and traceability: Store prompts, outputs, and reviewer sign-offs for every deliverable. Make audits easy.
- Incident playbook: Define steps for retracting, correcting, and communicating if an AI-assisted report is found inaccurate.
- Legal and compliance checks: For regulated topics, require legal review and documented source trails.
Operating metrics that keep you honest
- Hallucination rate: percent of sampled outputs with factual or citation errors.
- Citation validity: share of references with verifiable sources and live links/DOIs.
- Time-to-detect and time-to-correct: from issue report to published fix.
- Policy adherence: percent of AI uses with logged prompts and approvals.
- Cost of rework: hours and dollars spent correcting AI-assisted deliverables.
Context from recent cases
- Apple paused an AI news summarization feature after users reported false outputs.
- A U.S. federal judge sanctioned attorneys for filing a brief with fabricated case citations generated by an AI tool.
Expect more incidents. Adoption won't stop, but tolerance for sloppy controls will. Treat AI like a junior analyst: helpful, fast, and fallible - and always supervised.
Action for CFOs and finance leaders
- Issue a firmwide AI usage policy and require training for all finance staff.
- Stand up a small AI risk committee that owns standards, tools, and audits.
- Prioritize use cases with clear source data (e.g., reconciliations, variance notes) before policy-heavy or legal topics.
- Tie compensation or OKRs to accuracy, not just speed.
If you're formalizing your team's approach to safe adoption, here's a practical resource to explore AI options for finance use cases: AI tools for Finance.
Your membership also unlocks:
