oMLX is a native macOS inference server designed to optimize Large Language Model (LLM) inference on Apple Silicon. It is built upon MLX, Apple's machine learning framework for Macs, and was developed by jundot. The primary goal of oMLX is to provide a highly efficient and responsive local AI experience, particularly for demanding applications like coding agents, by addressing common performance bottlenecks associated with LLM inference on local hardware.

The tool functions by providing a local inference server on macOS that can run various LLM, VLM, embedding, and reranker models. It aims to dramatically reduce the "Time To First Token" (TTFT) for agentic workloads by employing a unique caching mechanism. Instead of discarding the Key-Value (KV) cache when context shifts, oMLX persists these cache blocks to SSD, allowing for rapid retrieval and reuse across requests and server restarts. This architecture also supports continuous batching for improved throughput when handling multiple concurrent requests. It exposes OpenAI and Anthropic-compatible API endpoints, making it a drop-in replacement for many existing AI clients and tools.

Some of the key features are:

• Paged SSD KV caching: Cache blocks are persisted to disk in safetensors format using a two-tier architecture where hot blocks remain in RAM and cold blocks move to SSD with an LRU policy. Previously seen prefixes are restored across requests and server restarts, avoiding recomputation.
• Continuous batching: The server handles concurrent requests efficiently using mlx-lm's BatchGenerator, offering significant generation speedups at higher concurrency levels by eliminating queuing for single requests.
• Native macOS menu bar app: Users can start, stop, and monitor the server directly from a dedicated macOS menu bar application. This app includes a web dashboard for managing models, chatting, and viewing real-time metrics. It is signed, notarized, and supports in-app auto-updates, built without Electron.
• Multi-model serving: oMLX can load and serve multiple types of models concurrently, including LLM, VLM (Vision-Language Model), embedding, and reranker models. It implements LRU eviction to manage memory when resources become low, and allows users to browse and download models directly from its admin dashboard.
• OpenAI + Anthropic drop-in: The server provides API endpoints compatible with both OpenAI (/v1/chat/completions) and Anthropic (/v1/messages) specifications. This enables seamless integration with clients and tools designed for these APIs, such as Claude Code, OpenClaw, and Cursor.
• Tool calling + MCP: oMLX supports a wide array of major tool calling formats, including JSON, Qwen, Gemma, GLM, and MiniMax. It also integrates MCP (Model-Controlled Program) tools and features tool result trimming for handling oversized outputs, with configurations available per model.

oMLX operates as a background service on macOS, accessible and managed through a native menu bar application. Users first install the DMG or build from source, then configure their model directory. The server can reuse existing LM Studio model folders or download new MLX-format models from HuggingFace via its web dashboard. Once running, it exposes a local API endpoint (typically localhost:8000) that client applications can connect to. When a request comes in, oMLX processes it using its optimized MLX backend, leveraging paged SSD KV caching to significantly speed up responses, especially for conversational or agentic tasks that frequently revisit previous contexts. The continuous batching mechanism allows multiple concurrent requests to be processed efficiently without waiting.

Some common use cases include:

• Accelerating coding agents: Drastically reducing the TTFT for coding assistants like Claude Code, OpenClaw, and Cursor, which frequently invalidate and reuse context during development workflows.
• Local LLM development and experimentation: Providing a high-performance local environment for developers to test and iterate on LLMs and other AI models directly on their Apple Silicon Macs.
• Multi-model AI applications: Running various types of AI models simultaneously, such as a large language model, a vision-language model, and an embedding model, for complex local AI applications that require diverse capabilities.
• Benchmarking and performance analysis: Utilizing the built-in performance explorer and community benchmarks to evaluate and compare the performance of different models and configurations on Apple Silicon hardware.
• Integration with existing AI clients: Acting as a drop-in local backend for any application or tool that supports OpenAI or Anthropic API specifications, enabling privacy-focused and low-latency AI interactions.