Sup AI

About Sup AI

Sup AI is an ensemble AI service that runs multiple large language models in parallel and synthesizes their outputs by measuring per-token confidence. It downweights segments with high entropy (uncertain outputs) and amplifies low-entropy segments to reduce the incidence of hallucinations. The system scored 52.15% on Humanity's Last Exam, which is 7.41 points higher than the best individual model in the same evaluation.

Review

Sup AI tackles a common problem with LLMs by combining many models and weighting their outputs according to confidence signals. The approach is backed by published evaluation data and openly shared methodology, and the product offers a pragmatic onboarding path with a modest starter credit.

Key Features

• Ensemble inference: runs multiple LLMs (from a pool of up to 339) in parallel and synthesizes a single response.
• Entropy-based weighting: uses token probability distributions to identify high- and low-confidence segments and adjusts output contributions accordingly.
• Deterministic checks: the synthesizer can incorporate code execution or web search to verify or augment model outputs when appropriate.
• Cost and latency optimizations: a compaction algorithm and prompt caching reduce token and runtime overhead compared with a naive ensemble implementation.
• Open evaluation and research: methodology, eval code, and raw results are available for inspection (white paper, GitHub repo).

Pricing and Value

Onboarding requires card verification and a $10 starter credit (no auto-charge). There is a limited-time 20% discount for the first month in some campaigns. The value proposition is clearer for users who prioritize lower hallucination rates and transparent evaluation over the lowest possible cost: ensemble results can outperform individual models, and internal optimizations reduce but do not eliminate the extra token/compute cost associated with running multiple models.

Pros

• Measurable accuracy improvement in published evaluations (52.15% on Humanity's Last Exam versus 44.74% for the best individual model in the same run).
• Entropy-based synthesis offers a principled way to reduce hallucinated content compared with single-model outputs.
• Flexible synthesizer can use deterministic checks (code, web search) to validate critical outputs.
• Transparent research and shared evaluation artifacts help users assess claims and reproduce results.
• Simple trial setup with starter credit and no automatic charges.

Cons

• Overall accuracy remains far from perfect (52.15% indicates substantial room for improvement for many tasks).
• Ensemble inference increases token and compute use; while optimizations reduce overhead, costs can still be higher than single-model options.
• Not all provider APIs expose token logprobs, so confidence estimation sometimes relies on heuristics or approximations.

Sup AI is well suited for teams and individuals who need cleaner, more reliable natural language responses than a single model can typically provide-for example, for exploratory research, sensitive document review, or apps where reducing hallucinations is important. It may be less appropriate for cost-constrained projects that require deterministic numeric accuracy without external verification. For more details or to try it, visit sup.ai.

Open 'Sup AI' Website