ARBR

About ARBR

ARBR is an open-source, self-hosted control layer that sits between applications and AI models. It presents an OpenAI-compatible endpoint so teams can route, govern, observe, evaluate, and deploy across multiple providers without changing existing code. The tool is MIT-licensed and provider-neutral, built to run as a standalone gateway or on top of LiteLLM.

Review

ARBR tackles a specific production problem: knowing which workloads can safely move to a different model, with evidence to back the decision, and confirming the result after rollout. It observes traffic patterns, surfaces model-switching opportunities, and builds evaluation datasets from representative requests. The routing logic follows explicit, auditable rules - models that are explicitly pinned stay pinned, while workloads marked "auto" follow only the policies the team has enabled.

Key Features

• Observes live workloads and identifies which ones are candidates for switching to a different model
• Builds evaluation datasets from actual traffic and compares candidate models across quality, cost, latency, format adherence, and critical failures
• Supports a staged deployment pipeline: shadow testing against mirrored traffic, guarded canary with automatic rollback, and human-promoted full rollout
• Automatic canary monitoring that recomputes candidate-vs-baseline metrics every 5 minutes over a trailing 60-minute window, with rollback triggered on guardrail breaches
• Demo mode that lets users explore the full workflow without adding a provider key

Pricing and Value

ARBR is free and open source under the MIT License. The team has not published any paid tiers, enterprise plans, or managed hosting options. Self-hosting is the only deployment model described in the current documentation.

Pros

• Keeps the final model-change decision human-controlled, with an audit trail recording who promoted a swap
• Rollback triggers combine production telemetry (error rate, p95 latency, cost savings) with one eval-based metric, rather than relying solely on LLM-judged scores
• OpenAI-compatible endpoint means applications don't need rewrites to connect
• Works as a standalone gateway or layers above an existing LiteLLM setup
• MIT license imposes no restrictions on modification or commercial use

Cons

• Requires self-hosting, so teams without infrastructure to run Docker containers will find setup heavier than a managed service
• Provider integrations are limited to what LiteLLM supports; teams using niche or proprietary APIs may hit coverage gaps
• Not well suited for teams that don't run LLM workloads in production yet - the tool's evaluation and canary pipeline assumes live traffic to observe and compare

ARBR fits teams already running LLM workloads in production who want a structured, evidence-based process for model changes without vendor lock-in. The staged rollout pipeline gives operators room to test swaps on shadow and canary traffic before committing. Organizations that prefer a fully managed SaaS control plane, or that don't have production traffic to observe yet, won't find those use cases addressed in the current release.

Open 'ARBR' Website