AI news ·
How to use decision models with Amazon Bedrock AgentCore Evaluations
Amazon Bedrock AgentCore Evaluations now supports decision models as custom evaluators that return calibrated probability scores instead of free-form text. A cascade pattern runs these cheap, narrow checks on every turn, then escalates only ambiguous sessions to full LLM evaluators for review.
Amazon Bedrock AgentCore Evaluations now supports decision models as custom evaluators, giving operations, IT, and product teams a faster, more consistent way to score agent responses. The new capability lets you run narrow, repeatable checks-such as whether a response stayed on policy or a tool call is safe-using small, specialized models that return calibrated probability scores instead of free-form text.
AgentCore Evaluations scores an agent through its entire lifecycle: on demand during development, in batch jobs against stored sessions, and online against production traffic. Alongside 15 built-in evaluators, it supports custom evaluators defined as either an LLM-as-a-judge or a code-based evaluator. Code-based evaluators run your own logic in an AWS Lambda function, and that logic can call any external system, model, or API, including a decision model.
What a decision model does differently
A decision model is a small, specialized model trained to select from a fixed set of options and return calibrated confidence scores. A general-purpose LLM evaluator produces a score with a rationale. A decision model produces a probability distribution over predefined answers. That gives you a direct confidence measure you can threshold, aggregate, and act on programmatically.
Decision models suit questions with a fixed set of answers that an informed person could decide at a glance. "Is every fact in this response supported by the tool results?" is a yes/no question. "How helpful was this response, on a four-level rubric?" is a score question. Both fit the model's design.
Two backends, same integration pattern
The sample architecture connects two decision model backends to AgentCore Evaluations through code-based evaluators. The first is Jev, a hosted decision model from TypeSafe. The second is Strands Decider 2B, an open-source decision model you run yourself, either locally for testing or on Amazon EC2 inside your VPC.
Both backends support three question types. A noul question poses a yes/no question and returns a probability between 0 (no) and 1 (yes). A choice question asks which of up to 255 options applies, returning the most probable option along with the full distribution. A score question asks which of 2 to 10 ordered levels applies, returning a probability-weighted score.
The Lambda function registered as the evaluator receives OpenTelemetry spans of the agent, normalizes the data, and formats it for the decision model. For trace-level evaluation, only the targeted turn and brief prior history are sent. For session-level evaluation, the full conversation is included. The adapter then converts the model's probability distribution into a result AgentCore expects. For a noul question, the probability of true becomes the value. For a score question, the value is the probability-weighted expected level scaled to 0-1.
Validating a decision-model evaluator on your own traffic
When you rewrite a rubric as a decision model question, you have created a new measurement, not a copy of the old one. A reframed rubric often lands stricter or looser than intended, and you will not see that drift until you check it against real sessions. The validation procedure has five steps.
First, build a labeled sample from stored sessions with enough coverage of the failures and edge cases you care about. Second, have three independent labelers score each turn and take the majority vote as the reference label. Keep track of which turns were unanimous and which were split-the split-vote turns reveal ambiguity in the rubric itself.
Third, run both the decision-model evaluator and the matching built-in evaluator on the same inputs in a single batch evaluation. Fourth, read the results through three lenses: agreement rate with human labels, calibration of confidence scores, and disagreement analysis between the two evaluators. Most disagreements trace back to the wording of the question, and a revised question clears them up. Fifth, treat this as a repeatable check. Any time you reword a question or move to a new model version, run it again.
The cascade pattern for production
The built-in evaluators remain the foundation. They assess the response against a rubric and its supporting context, and explain the judgment so developers can investigate what went wrong. Decision models work best as a screening layer, not a replacement.
The cascade pattern runs a fast, calibrated decision model on every turn of every session to get a cheap pass/fail signal. Then it invokes a full LLM evaluator only on sessions where the signal falls below your threshold. As the JEV-as-a-Judge preprint describes it, the distinction is "reading versus deriving." A narrow decision-model check flags a turn for attention, while the built-in evaluator provides the broader assessment and reasoning needed to review it.
For production, the recommendation is to configure online evaluation with the built-in evaluators you rely on, add decision-model evaluators for domain-specific yes/no and choice questions, and add deterministic code-based evaluators for anything you can compute exactly. Each week, query the online results for sessions where a decision-model check failed or scored near its threshold, and run batch evaluations on those sessions with the built-in evaluators to get explanations.
Why this matters for operations and product teams
Decision-model evaluators give you a measurement you can threshold with confidence, which matters when you need to gate a release or trigger an automated workflow. Because a single request can answer several questions about the same state, you can screen every turn cheaply and escalate only the ambiguous ones for deeper review. The sample code in the AgentCore samples repository deploys the full pipeline-HR Assistant agent, deterministic checks, built-in evaluators, and decision-model evaluators-so you can test the framework against your own rubrics before committing it to production traffic. Professionals evaluating agent quality at scale may also find structured training on evaluation methods useful through AI Quality Assurance Courses.