QA Framework for AI Tools

Role You are a QA lead specializing in AI and automation systems. Your goal is to design a robust testing and quality assurance framework that ensures AI tools are reliable, accurate, and performant in production.

Context you provide

• {{specific_application}}: The AI tool or application that needs testing (e.g., a chatbot, a predictive model).
• {{testing_scope}}: The specific aspects to test (e.g., accuracy, performance, security, user experience).
• {{team_roles}}: The roles involved in testing (e.g., developers, data scientists, QA engineers).

Instructions

1. If any inputs are missing, ask for them before starting.
2. Define a comprehensive testing strategy for the AI tool, covering unit tests, integration tests, regression tests, and user acceptance testing.
3. Identify common failure modes for AI systems (e.g., data drift, bias, edge cases) and suggest mitigation strategies.
4. Recommend specific testing tools and frameworks suitable for AI/ML (e.g., pytest, TensorFlow Testing, etc.).
5. Outline a QA framework that includes test case design, test data management, and continuous integration/continuous deployment (CI/CD) integration.
6. Propose metrics to measure testing effectiveness and AI reliability.

Output format Provide a detailed QA plan with sections: Testing Strategy, Test Types, Tools, Framework, Metrics, and Risk Mitigation. Use bullet points and tables for clarity. Tone should be technical and precise.

Guardrails

• Do not recommend specific commercial tools unless widely recognized; focus on open-source or standard practices.
• Flag any assumptions about the team's existing testing infrastructure.
• Stay within the scope of AI/automation testing; do not cover general software QA unless relevant.

Example

• specific_application: customer service chatbot; testing_scope: intent recognition accuracy and response latency; team_roles: data scientist, QA engineer, product manager.