oMLX

About oMLX

oMLX is a Mac LLM inference server that runs from the menu bar. It serves text, vision, OCR, embedding, and reranker models with continuous batching, and it includes a RAM+SSD tiered KV cache that survives restarts. The tool exposes OpenAI and Anthropic compatible APIs, is built with native Swift rather than Electron, and is open source under the Apache 2.0 license.

Review

oMLX launched this week and targets a specific pain point for developers running agentic coding loops on Apple silicon. The core claim is straightforward: Claude Code and Cursor respond in about 5 seconds instead of 90 seconds because the tool persists the KV cache to disk. That's a meaningful difference from most local model setups, which discard the whole conversation context on every turn and recompute it from scratch.

The project has been in active development since February, with a public commit history that shows near-daily shipping. It currently has 21,000 stars on GitHub and is free to use.

Key Features

• Menu bar LLM inference server that turns a Mac into a full inference endpoint
• RAM+SSD tiered KV cache that persists across restarts, so old context returns in milliseconds instead of being recomputed
• Continuous batching for text, vision, OCR, embedding, and reranker models
• OpenAI and Anthropic compatible APIs for drop-in integration with existing tools
• Native Swift implementation, not an Electron wrapper

Pricing and Value

oMLX is free and open source under the Apache 2.0 license. There is no paid tier or subscription model mentioned in the launch materials. The value case is about removing hardware constraints: the SSD tier means you are limited by available disk space rather than the RAM you bought up front.

Pros

• KV cache persistence eliminates the 90-second recompute penalty in agent loops, cutting wait times to roughly 5 seconds
• Runs entirely on your own machine, so data never leaves the building
• API compatibility with OpenAI and Anthropic means existing tooling works without modification
• Apache 2.0 license permits commercial use and modification
• Active development cadence with daily commits since February

Cons

• Cache size scales with context length times model size, so long agent sessions on mid-sized models consume gigabytes of SSD space; loading two models at once turns disk management into an active budgeting exercise
• There is no warning when the disk cache fills up - performance degrades gradually, with time to first token stretching while tokens per second stays normal, which can be mistaken for a model quality issue
• Not well suited for users who only run occasional one-off chat queries, where tools like LM Studio already handle the simpler use case without the added complexity of cache management

oMLX fits developers running agentic coding loops on Apple silicon who want local inference without the prefill penalty on every turn. It also suits studios handling unreleased creative work where data privacy matters more than raw speed. If your workflow is single-turn chat or you lack the disk space for multi-gigabyte caches, the tool's main advantage won't apply to you.