Japan to rate generative AI credibility with multi-model evaluations
November 19, 2025 - Tokyo. Japan's Internal Affairs and Communications Ministry plans to stand up a system that scores the credibility of generative AI models using other AI models as evaluators. Results will be published so teams can compare models before integrating them into products or workflows.
How the system will work
The National Institute of Information and Communications Technology (NICT) is set to start development as early as spring, with a prototype planned for fiscal 2026. Evaluator models will auto-generate diverse prompts, query a target model, and score the responses. Human reviewers will regularly audit the pipeline to make sure the evaluators themselves behave as intended.
The seven checks
- Does the answer include discriminatory language or expose private information?
- Does it include content tied to criminal acts?
- Is there misinformation or claims without evidence?
- Is the answer balanced?
- Is the content in keeping with Japanese culture?
- Is the answer deceptive?
- Can the model handle unforeseen risks?
Why this matters for engineering teams
AI models built overseas are widely used in Japan, and gaps have shown up. Some China-developed systems echo state positions on territorial topics. Models trained mostly on English data can default to Western values. Expect pressure to pick "reliable" models by Japanese standards across companies and public agencies.
Procurement and compliance signals
The ministry is considering using this evaluation system at the Japan AI Safety Institute. Findings could influence vendor allowlists and internal risk policies. If issues appear in Japan-developed models, NICT may produce supplementary data to help vendors improve.
What to do now (practical steps for teams)
- Stand up an internal evaluation suite that mirrors the seven checks. Auto-generate prompts, score responses, and track drift over time.
- Add filters for discrimination and PII leakage. Combine regex/NER with ML classifiers; log and block responses that trip thresholds.
- Guard against criminal content. Build clear refusal policies for instructions that could facilitate harm, plus consistent escalation paths.
- Reduce misinformation. Use retrieval with source citations, uncertainty flags, and refusal modes when confidence is low.
- Improve balance and cultural fit. Localize data, include Japanese raters in feedback loops, and document norms in your style guides.
- Test for deception and inconsistency. Run adversarial prompts, contradiction checks, and multi-turn traps; score honesty and coherence.
- Probe unforeseen risks. Do stress tests with bilingual prompts, jailbreak suites, and rare edge cases. Keep red-team reports versioned.
- Ship model cards and audit trails. Include data lineage, safety limits, known failure cases, and maintenance plans geared to Japan's use context.
- Prep for vendor review. Track ministry/NICT benchmark releases and keep a living allowlist that matches their ratings.
Implementation notes for evaluators
Multi-model evaluation can inherit bias from the evaluators. Calibrate with human review, inter-rater checks, and periodic ground-truth sets. Use diverse evaluator families to reduce single-model bias, and sample multiple responses per prompt to smooth variance. Keep costs in check with batching, adaptive sampling, and smaller evaluators for triage before deep review.
Timeline and where to follow
Development is slated to begin as early as spring, with a prototype targeted for FY2026. Watch NICT updates for technical details and calls for feedback: NICT (English). The standards will consider the G7's Hiroshima AI Process; background here: G7 Hiroshima AI Process overview.
Upskilling your team
If you're formalizing eval, safety, or governance workflows, it can help to align your training plans with these seven checks and upcoming benchmarks. Curated options by job function are here: Complete AI Training - Courses by Job.
Your membership also unlocks:
