States Must Establish AI Safety Standards Before Schools Deploy Unproven Systems
Artificial intelligence tools are entering K-12 schools faster than oversight mechanisms can track them. These systems now influence student discipline, grades, placement decisions, attendance monitoring, and safety protocols-yet most lack transparent testing, civil rights review, or consistent safeguards.
The result is material legal and operational risk. AI systems that affect discipline or eligibility may violate Title VI, Title IX, Section 504, and the Individuals with Disabilities Education Act. Tools that collect student data raise compliance questions under the Family Educational Rights and Privacy Act. Meanwhile, most districts lack the capacity to evaluate vendor claims or negotiate contracts that protect against bias and privacy breaches.
States can address these gaps using procurement authority they already possess. A structured framework-combining vendor disclosure requirements, pre-deployment review, performance monitoring, and leadership training-can reduce discrimination risk while allowing responsible innovation.
The Problem: Speed Without Safeguards
Recent incidents show what happens when AI enters schools without adequate oversight. Los Angeles Unified School District contracted with an education-tech startup that misused student data. A Baltimore County school's AI security camera system misidentified a bag of chips as a firearm, triggering unnecessary alarm and potential law enforcement response.
These are not edge cases. AI systems can shape high-stakes decisions-who gets suspended, which students enter gifted programs, who faces law enforcement referral-with limited transparency or accountability. The core problem is that procurement systems designed for conventional software cannot handle probabilistic tools that adapt over time and influence consequential outcomes.
Decentralized purchasing compounds the issue. Thousands of districts negotiate independently with vendors, often lacking expertise to assess accuracy claims or data security practices. Smaller districts are particularly vulnerable. Contracts frequently limit audit rights, restrict data deletion, and create vendor lock-in.
Some AI use in schools may genuinely help. Tutoring systems that identify students needing support, tools that expand language accessibility, and systems that streamline administrative tasks can add value. The question is not whether to adopt AI, but how to adopt it responsibly.
Recommendation 1: Establish Statewide Procurement Guardrails
States should classify AI systems by risk level. High-risk systems include those affecting discipline, expulsion, placement in gifted or special education, grading, graduation eligibility, behavioral monitoring, threat assessment, and facial recognition.
These uses are high-risk because they directly affect educational access, safety, and civil rights protections under federal law. They warrant proportionate oversight before purchase.
Require a pre-purchase checklist. Before districts buy high-risk AI, they should document the system's purpose, which student populations it affects, error rates, whether it has been tested for disparate impact across racial groups and students with disabilities, who reviews outputs, and how students or parents can appeal.
Vendors should disclose training data sources, subgroup performance metrics, data retention policies, subcontractor lists, server locations, and cybersecurity certifications. These requirements do not demand disclosure of proprietary algorithms but do require sufficient transparency for schools to conduct due diligence.
Embed standard contract clauses. States should develop model provisions that districts adopt. Core clauses should address data minimization, security and breach notification, audit rights, termination and data deletion, and restrictions on subcontractors and offshore data transfers.
Recommendation 2: Require Algorithmic Impact Assessments Before Deployment
States should require districts to complete and publicly post an Algorithmic Impact Assessment before deploying any high-risk AI system. An AIA is a structured evaluation of purpose, risks, legal implications, and mitigation strategies.
AIAs do not ban AI use. They shift oversight upstream, identifying civil rights and safety risks before systems affect students rather than responding after harm occurs.
A mandatory AIA should include the system's purpose and which students are affected; data inputs and whether protected characteristics or proxies are used; accuracy and validation results; disparate impact testing results; human oversight mechanisms; due process protections; and privacy safeguards.
States should require a two-tiered transparency approach. A public version includes narrative descriptions, performance metrics, limitations, demographic testing results, and a plain-language summary for parents. A regulator-facing appendix contains more detailed technical documentation.
Raw datasets and proprietary algorithms need not be disclosed. But vague assurances-"tested for bias"-are insufficient. AIAs must include independent testing, documented metrics, and clear methodologies.
Recommendation 3: Prohibit Predictive Discipline and Law-Enforcement-Derived Systems
Certain AI uses pose risks that procurement safeguards alone cannot address. Systems that predict future misconduct or generate behavioral threat scores risk replicating historical patterns of bias and increasing unnecessary discipline and law enforcement involvement, particularly for students of color, students with disabilities, and LGBTQ+ students.
States should prohibit AI systems that generate forward-looking risk scores for misconduct to justify discipline; produce behavioral threat scores like "aggression" absent specific evidence; rely primarily on law enforcement datasets; use facial recognition; identify objects as weapons; or integrate student data into external law enforcement systems.
Predictive discipline tools are especially problematic. Because school discipline data reflect documented racial disparities-Black students are disproportionately disciplined for similar behaviors-the same algorithmic risk score may lead to different interventions depending on student race, exacerbating existing inequities. Systems that aggregate attendance, prior discipline, or behavioral indicators risk institutionalizing biased baselines.
Law enforcement-derived analytics introduce additional risks. Police datasets often reflect patterns of over-policing. Incorporating them into school decision-making imports external bias into educational settings. Schools are educational institutions, not extensions of the criminal justice system.
Narrow exceptions may apply only with heightened safeguards: AI outputs cannot be the sole basis for adverse action, trained personnel must review all outputs, and systems must undergo annual disparate impact analysis.
Recommendation 4: Require Ongoing Monitoring and Incident Response
AI systems evolve and degrade over time. Effective governance requires continuous monitoring, not one-time approval.
States should require districts to submit annual public reports on system performance using a standardized template. Reports should include error rates, misclassification trends, demographic disparities in flagging rates and disciplinary actions, frequency of human overrides, and complaint volume and resolution.
Districts should adopt rapid incident response protocols for significant harms: major data breaches, unsafe outputs triggering law enforcement, systemic bias, and widespread false positives. Protocols should include immediate containment, notification to families within 48-72 hours, root cause analysis, and corrective action plans.
Districts should not hesitate to pause or suspend systems when student safety or civil rights are at stake. In high-stakes environments, it is prudent to intervene rather than wait.
Authorization should automatically sunset after three years unless renewed based on demonstrated accuracy, absence of unexplained demographic disparities, documented educational benefit, and compliance with reporting and audit requirements.
Recommendation 5: Build State Technical Assistance and Vendor Accountability
Procurement guardrails will only succeed if districts have capacity to implement them. Many districts, especially rural ones, lack expertise in AI evaluation, data governance, and contract negotiation.
States should establish targeted training for procurement staff, technology leaders, and administrators. Training should cover risk classification, AIA completion, evaluation of vendor claims, disparate impact analysis, contract negotiation, and incident response obligations. Delivery can leverage existing professional development structures: webinars, regional workshops, online modules.
States may establish an optional pre-vetted "approved vendor" pathway. Vendors voluntarily submit documentation demonstrating compliance with state disclosure, testing, and contract requirements. The state conducts structured review. If approved, vendors are listed in a public registry. Districts may still procure other vendors but must complete full independent review.
States should mandate independent evaluation of high-risk systems through secure data enclaves, aggregated performance reviews under confidentiality agreements, or partnerships with public universities for validation studies. Independent evaluation helps test vendor claims, detect disparate impacts, and build public trust.
Recommendation 6: Build Leadership Capacity for District Administrators
Procurement reform fails without leadership capacity. Superintendents and senior administrators often make AI adoption decisions based on vendor presentations yet may lack training in algorithmic risk, civil rights, and technology contract governance.
States should establish targeted AI governance training for superintendents, chief technology officers, chief academic officers, and school board members. Core components should cover civil rights risks under Title VI, IDEA, Section 504, and FERPA; procurement literacy including avoiding vendor lock-in; how to distinguish supportive from punitive AI use cases; and crisis preparedness for discriminatory or unsafe outcomes.
States can integrate this training into existing professional development: annual superintendent conferences, certification renewals, school board association trainings, and regional education service agency programs.
When superintendents understand the governance framework, they are more likely to demand compliance with procurement guardrails, resist premature adoption, dedicate staff time to meaningful review, and support transparency. AI governance is cross-functional-superintendents must coordinate legal, procurement, technical, and ethical considerations rather than treat AI as purely a technical issue for IT staff.
Why This Framework Works
These six recommendations rely on authority states already possess: procurement standards, contracting requirements, and oversight of local education agencies. They address civil rights and privacy risks upstream, before litigation or public controversy. They reduce duplication and strengthen districts' negotiating leverage with vendors. Uniform standards lower compliance costs and incentivize vendors to compete on transparency and fairness testing rather than marketing claims.
Modest investments in technical assistance and leadership training can create clear, workable standards for districts and predictable expectations for vendors. The result is safer technology deployment, lower discrimination risk, stronger data governance, better stewardship of public funds, and greater public trust.
AI in schools should expand opportunity, not erode it. Acting now, while adoption norms are still forming, allows education leaders to ensure innovation and student rights advance together.
Learn more about responsible AI implementation in education: Explore AI for Education resources, or develop leadership skills with the AI Learning Path for School Principals.
Your membership also unlocks:
