Explaining the unexplainable: Why AI in finance must earn our trust
AI is everywhere in finance: credit risk, underwriting, fraud, portfolio construction, research. As models get smarter, they also get harder to explain. That's a problem for trust, accountability, and compliance.
In the US, explainability isn't optional. Banking regulators expect AI and machine learning to fit within model risk management. The CFPB requires "specific and accurate reasons" for adverse actions, even when the system is complex. The SEC has raised flags on conflicts when broker-dealers use predictive analytics with retail investors.
The black-box risk is real-and costly
Finance teams can't defend what they can't explain. Lack of explainability is a top barrier to AI adoption across investment roles. A major culprit is weak data infrastructure: poor quality, limited access, and thin governance. When data lineage is murky, auditability and traceability break down.
The risk multiplies in credit. Deep models trained on alternative data (transactions, behavior, device signals) can correlate with protected attributes by proxy. You may meet a backtest and still drift into discrimination without clear reasoning trails.
Private credit and deal evaluation face the same issue. If training data bakes in bias, strategy can drift, allocation can skew, and the "why" behind positions goes missing.
Different stakeholders, different explanations
One explanation doesn't fit all. Map explanations to the person making a decision off them.
- Regulators: Documentation, data lineage, audit logs, challenger results, adverse action reason codes, stability under stress.
- Portfolio managers: Sensitivity to drivers, scenario paths, regime behavior, drift dashboards, feature importance over time.
- Risk teams: Out-of-sample performance, fairness tests, stability during stress, limits and overrides, model change control.
- Customers: Plain-language reasons, counterfactuals ("If income were $5,000 higher, approval"), and clear next steps.
Two paths to explainability
- Ante-hoc (interpretable by design): Decision trees, scorecards, rules, generalized additive models. You trade some accuracy for clarity. In highly regulated decisions, that trade can be worth it.
- Post-hoc (explain a trained model): SHAP for feature contribution, LIME for local explanations, partial dependence and ICE plots for response curves, heatmaps for attention, and counterfactuals for "what it would take" to change an outcome. Useful in fast decisions like trading and fraud.
Where explainability can mislead
Explanations can be wrong-or just persuasive. People tend to trust neat charts and reason codes even when the logic is thin. Different tools can also disagree on what "drove" a prediction. That inconsistency makes standards hard to set across firms and jurisdictions.
There's also no universal benchmark for explanation quality. "Looks good" isn't a control. You need tests for stability, completeness, and fairness, or you're flying blind.
What to do now: A practical playbook
- Codify governance
- Maintain a live model inventory with purpose, owners, data sources, limitations, and risk tier.
- Document data lineage, access controls, and transformations end to end.
- Map each model to required disclosures (e.g., adverse action reasons) and reviewer checklists.
- Build stakeholder-specific explanations
- Regulators: audit trails, versioned artifacts, reason-code libraries tied to features, and evidence of challenger tests.
- PMs: sensitivity to top drivers, scenario outcomes, and regime-switch behavior.
- Risk: fairness metrics, stability under stress, performance by segment, alerting on drift.
- Customers: plain-language summaries and counterfactuals that are actionable and respectful.
- Invest in data foundations
- Quality checks at ingestion, standardized schemas, and reference data controls.
- Feature stores with lineage and permissions.
- Retention rules aligned to regulation and model retraining needs.
- Move to real-time explainability where impact is high
- Pre-compute reason codes and SHAP summaries for decision-time responses.
- Set latency budgets that include explanation generation.
- Log decision context, explanation artifacts, and overrides for audit.
- Test explanation quality
- Check explanation stability across model versions and data slices.
- Compare multiple XAI methods and resolve conflicts before production.
- Run counterfactual fairness tests to catch proxy discrimination.
- Keep a human in the loop where stakes are high
- Define clear escalation paths and override rights.
- Train staff to question explanations instead of rubber-stamping them.
- Track outcomes after overrides to refine policy and models.
Regulatory anchors worth noting
Two references clarify expectations on explanations and governance:
- CFPB guidance on adverse action notices with complex models: Circular 2023-03
- Federal Reserve guidance on model risk management: SR 11-7
Bottom line
Explainability isn't a checkbox. It's the difference between defensible finance and guesswork wrapped in math. If you can't explain it, you can't govern it-and you won't earn trust from regulators, clients, or your own team.
Build for clarity from day one. Then prove it in production.
Further help
If your team is building or auditing AI in finance and needs practical training on governance and explainability, explore curated resources: AI tools for finance.
Your membership also unlocks:
