Prefactor

About Prefactor

Prefactor is an evaluation layer for AI agents running in production. It scores every agent run in real time, detects quality regressions and drift as they happen, and gives engineering teams a live view of how their agents perform at scale. The tool targets teams that ship agents to customers and need to close the gap between passing evals and actual production behavior.

Review

Most agent evaluation stops at pre-deployment checks. Prefactor focuses on what happens after shipping - it watches every run, scores it, and can intervene automatically when a run crosses a defined boundary. The system connects to your agent runtimes, ingests spans, and lets you define evals that run on each interaction without requiring an LLM call by default.

Key Features

• Real-time run scoring with deterministic risk profiles. Prefactor builds a risk profile from the agent's schema - what tools it can access, what data it can touch - and compares actual behavior against that profile. No model client is involved, so these checks cost zero tokens.
• Custom evals grounded in external context. You can define LLM-as-judge, technical checks, or qualitative metrics. Custom spans pull data from GitHub, Linear, Jira, or your database so evals reflect what actually happened, not just the transcript.
• Runtime enforcement actions. When a run violates a threshold, Prefactor can hold, approve, or block it automatically. Decisions are logged and enforced through the SDK or API. A killswitch can stop a run mid-breach, and you can insert a human-in-the-loop step at any point in the workflow.
• Agent lifecycle management. Agents can be versioned, staged, and promoted from dev to staging to production only when evals pass. Instant rollback is available when a promoted agent fails in production.
• Native framework support and OpenTelemetry compatibility. The SDK (TypeScript, Python) works directly with LangChain, Claude, Vercel AI, OpenClaw, and LiveKit. Other runtimes connect through OpenTelemetry or the core SDK.

Pricing and Value

Prefactor's free tier includes 25,000 spans per month with no credit card required. During the launch week, new users who set up their first agent get 1,000,000 free agent steps, valid until Friday 11:59pm PT. Paid plans beyond the free tier are not detailed in the current public information.

Pros

• Scores 100% of production traffic without defaulting to LLM-as-judge, avoiding per-run token costs for basic behavioral checks.
• Ingests spans from multiple frameworks and lets teams attach custom context from their own tools to each evaluation.
• Automated enforcement - hold, approve, or block - happens within the same run, not after a separate review cycle.
• Rollback and staged promotion workflows treat agents like versioned software artifacts.
• Setup takes one command (prefactor init) and traces the first run in under five minutes, according to the team.

Cons

• Per-turn latency scoring for voice agents is not a built-in quality dimension; teams must instrument timing as a custom span.
• Sub-agent evaluation treats each sub-agent as a separate instance - a breach in a nested agent does not automatically pause the parent run unless the user configures that stop condition manually.
• Prefactor is not well suited for teams that are only prototyping agents and not yet shipping to real users, as its value centers on production observability and enforcement.

Prefactor fits engineering teams that already have agents in front of customers and need a way to see quality drift and stop failures as they happen, rather than sampling traces after the fact. It will be less useful for early-stage experimentation where production traffic and risk are not yet present. The roadmap item to make evals first-class assets inside the platform could address some current configuration gaps, but that capability is not yet shipped.

Open 'Prefactor' Website