raullenchai/Rapid-MLX

The fastest local AI engine for Apple Silicon.
Drop-in OpenAI / Anthropic API · up to 3× Ollama's throughput (measured) · Runs on any M-series Mac.

_{rapidmlx.com · Docs · Model mirror · Desktop app · Discord}

---

Works with your AI stack

Use Rapid-MLX as a local backend for agents, apps, or your own code. If a client accepts an OpenAI- or Anthropic-compatible endpoint, it can usually use Rapid-MLX without an adapter.

Five Tier-1 agents are exercised end-to-end on real weights before release. See the tested compatibility matrix for exact API coverage and setup status.

---

Install

Desktop — macOS (Apple Silicon)

The easiest way to chat locally, manage models, and use vision, files, voice, and image generation from one app.

• Download Rapid-MLX Desktop
• Browse signed Desktop releases
• Requires an M-series Mac; Windows and Linux desktop builds are not available yet

CLI and server — macOS (Apple Silicon)

# Homebrew — prebuilt bottle from homebrew-core
brew install rapid-mlx
Or the guided installer — detects RAM and recommends a starter model
curl -fsSL https://rapidmlx.com/install.sh | bash

Both install the same rapid-mlx CLI. Prefer uv or pip, or want to verify the installer before running it? See alternative install methods and install security.

The guided installer prefers a runnable model already cached on this Mac when it fits the RAM tier. Otherwise its quick first-chat download is lfm2.5-1b-4bit below 16 GB and qwen3.5-4b-4bit at 16 GB or above; larger quality picks remain available through rapid-mlx recipe and the model picker.

---

Quick Start (60 seconds)

1. Chat with a model right now:

rapid-mlx chat

Defaults to qwen3.5-4b-4bit. First run downloads the weights (~3 GB) with a progress bar and drops you into a REPL. Type /help for slash commands, /exit to quit.

2. Or serve it for use from other apps:

rapid-mlx serve qwen3.5-4b-4bit

Starts an OpenAI-compatible HTTP server bound to http://localhost:8000. Point any client that supports a local custom endpoint (Aider, LangChain, OpenCode, PydanticAI, your own scripts) at http://localhost:8000/v1; Claude Code / Anthropic SDK uses http://localhost:8000 (the Anthropic messages route lives at /v1/messages under the same host).

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":"Say hello"}]}'

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
print(client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Say hello"}],
).choices[0].message.content)

3. Or wire up your coding agent — one command:

rapid-mlx launch claude-code

With a server running (step 2), this patches Claude Code's local config (~/.claude/settings.json) to route at http://localhost:8000 — no manual env vars, no editing JSON by hand. You get a fully local Claude Code: $0 per token, nothing leaves your Mac. Swap in cline or continue-dev for the other IDE clients, or run rapid-mlx launch list to see what's detected on this machine.

Cursor: Cursor currently routes BYOK requests through its own servers, so its servers cannot reach a Rapid-MLX endpoint on localhost. Rapid-MLX therefore does not generate a Cursor localhost config. If you intentionally expose the server through a public HTTPS tunnel, set RAPID_MLX_API_KEY=your-secret for both rapid-mlx serve ... and rapid-mlx launch cursor --server-url https://your-public-host. This is no longer a fully local connection; never expose an unauthenticated server. Rapid-MLX rejects explicit local/private addresses but cannot verify reachability from Cursor's network, whose DNS view may differ from your Mac.

Vision / audio / video / diffusion models? Base install is text-only (~460 MB). Vision, audio (TTS, STT, voice cloning), video generation, embeddings, and DFlash speculative decoding ship as opt-in extras. → Optional extras

Not into the terminal? Rapid-MLX Desktop bundles the same engine inside a one-click Mac app.

---

Image generation

Generate images locally from the Desktop Images tab or the OpenAI-compatible Images API. Install the image runtime when using the CLI:

pip install 'rapid-mlx[image]'
rapid-mlx serve flux2-klein-4b

curl http://localhost:8000/v1/images/generations \
  -H 'Content-Type: application/json' \
  -d '{"model":"flux2-klein-4b","prompt":"A red sailboat on an alpine lake","size":"1024x1024","seed":42}'

Edit an existing PNG or JPEG with the edit-specialized model:

In one terminal:

rapid-mlx serve qwen-image-edit

In another terminal:

curl http://localhost:8000/v1/images/edits \
  -F image=@input.png \
  -F model=qwen-image-edit \
  -F 'prompt=Change the sky to a warm sunset' \
  -F response_format=b64_json

flux2-klein-4b is the recommended starting point. Download sizes are the currently cataloged model payloads; minimum unified memory includes practical headroom for macOS and the serving process. Omit steps to use the validated family default shown below.

| Alias | Best fit | Modes | Download | Minimum unified memory | Default steps | | --- | --- | --- | ---: | ---: | ---: | | flux2-klein-4b | Recommended; fast everyday images | Generate + edit | 4.3 GiB | 12 GB | 4 | | bonsai-image-4b-2bit | Smallest download | Generate | 3.6 GiB | 12 GB | 4 | | z-image-turbo | Photorealistic images | Generate | 5.5 GiB | 16 GB | 8 | | flux-schnell | FLUX.1 compatibility | Generate | 9.0 GiB | 16 GB | 4 | | sdxl-base | SDXL compatibility | Generate | 6.5 GiB | 16 GB | 30 | | flux2-klein-4b-bf16 | Explicit full-precision Klein path | Generate + edit | 14.9 GiB | 32 GB | 4 | | hidream-o1-dev | Complex compositions | Generate | 16.4 GiB | 32 GB | 28 | | sd35-large-4bit | Stable Diffusion 3.5 | Generate | 15.3 GiB | 32 GB | 28 | | qwen-image | Text inside images | Generate | 28.9 GiB | 64 GB | 20 | | qwen-image-edit | Precise instruction edits and text changes | Edit | 34.9 GiB | 96 GB | 20 |

At 1024×1024 with the four-step Klein default, measured warm generation was about 9.2 seconds per image on an M3 Ultra. On a 32 GB M2 Pro, the q4 path measured a 52.7-second median; explicitly selecting flux2-klein-4b-bf16 reduced that median to 41.8 seconds (1.26× throughput). The eight-step z-image-turbo baseline was about 34 seconds on the M3 Ultra and 150 seconds on the M2 Pro. The q4 alias remains the predictable default; on the measured 32 GB M2 Pro, use the larger 14.9 GiB BF16 alias when latency matters. Exact results depend on resolution, power/thermal state, and other workloads. The server completion log reports total time and, when the backend exposes exact loop boundaries, measured denoise s/step. For the qualified 1024-square Klein shape it also reports estimated achieved TFLOPS; unavailable or unqualified values are never fabricated.

→ Reproducible precision benchmark and methodology

Image work is single-flight: one generation runs at a time so two diffusion pipelines cannot exhaust unified memory. Model weights retain their own licenses and use restrictions; review the upstream model card before commercial deployment.

→ Release dogfood coverage and reproducible acceptance contract

---

Video generation

Run text-to-video or image-to-video locally through the OpenAI-compatible Videos API. Three backends ship — Wan 2.1 / 2.2, CogVideoX-Fun and LTX-2.3 — across 8 registered checkpoints. wan2.2-ti2v-5b-q8 is the recommended starting point: smallest of the Wan set, and TI2V means one checkpoint does both text-to-video and image-to-video.

Requires Python 3.11+ (the video runtime does not support 3.10; core text and audio still do) and ffmpeg for the final MP4 mux.

pip install 'rapid-mlx[video]'
brew install ffmpeg
rapid-mlx serve wan2.2-ti2v-5b-q8

Create and download a clip:

curl http://localhost:8000/v1/videos \
  -F model=wan2.2-ti2v-5b-q8 \
  -F 'prompt=A fox running through fresh snow, cinematic tracking shot' \
  -F seconds=1 \
  -F size=832x512
Poll until GET /v1/videos/VIDEO_ID reports "status": "completed", then:
curl http://localhost:8000/v1/videos/VIDEO_ID/content -o output.mp4

The create call returns a job immediately. Poll GET /v1/videos/VIDEO_ID until status is completed. Add -F input_reference=@start.png for image-to-video.

Generation is serialized — one clip at a time — because two diffusion pipelines resident at once will exhaust unified memory. Expect minutes of compute per second of footage, not real time.

→ Every checkpoint, RAM requirement and tuning knob

---

Audio: speech, transcription, voice cloning

44 audio aliases behind the OpenAI-compatible /v1/audio/* endpoints — any OpenAI SDK works unchanged.

pip install 'rapid-mlx[audio]'
Text to speech
rapid-mlx serve kokoro
curl http://localhost:8000/v1/audio/speech \
  -H "Content-Type: application/json" \
  -d '{"model":"kokoro","input":"hello from rapid-mlx"}' --output hello.wav
Transcription (Whisper / Parakeet / SenseVoice)
rapid-mlx serve whisper-large-v3-turbo
curl http://localhost:8000/v1/audio/transcriptions \
  -F file=@hello.wav -F model=whisper-large-v3-turbo

Beyond the basics, three things you may not expect to run locally:

• Zero-shot voice cloning from a reference clip. indextts is the only

one that takes the clip alone; qwen3-tts-clone, f5-tts-zh and chatterbox all require ref_text (the clip's exact transcript) paired with ref_audio, and the request is rejected before generation if it is missing.

• Voice design — qwen3-tts-voicedesign has no named speakers at all.

Describe the voice you want in natural language via instructions (timbre, gender, age, accent, emotion, prosody) and it synthesises it.

• Forced alignment — qwen3-aligner takes audio *plus the transcript you

already have* and returns per-character timings. It never guesses at the words, so it cannot mis-hear them; that is what karaoke captions and beat-synced editing need.

Also: word-level timestamps on transcription, and local text-to-music at /v1/audio/music.

→ All 44 aliases across 13 families

---

Why Rapid-MLX

| | | |---|---| | Apple-Silicon-native | Pure MLX kernels — no llama.cpp fallback, no Metal shim. Continuous batching, prompt cache (radix + DeltaNet RNN snapshots), and a quantized live KV cache (int4/int8 on the continuous-batching cache + TurboQuant K8V4 codec) run at native MLX bandwidth on M1 → M4. | | Drop-in OpenAI / Anthropic API | /v1/chat/completions, /v1/responses (Codex CLI), /v1/messages (Anthropic SDK / Claude Code), /v1/embeddings, /v1/audio/*, /v1/videos — same wire as ChatGPT / Claude, no client adapter. | | First-class ecosystem coverage | 12 agent CLIs and 3 Python frameworks are wire-verified against real weights every release (5 are Tier-1, re-verified on current binaries) — Codex CLI, Claude Code, OpenCode, Qwen Code, OpenHands, Hermes Agent, Aider, Kilo Code, DeepSeek Harness, GitHub Copilot, Factory Droid, Moonshot Kimi Code + LangChain, PydanticAI, smolagents. |

→ Full feature breakdown

---

Use Cases

| | | | |---|---|---| | Chat in the terminal | rapid-mlx chat qwen3.5-9b-4bit | Streaming REPL, /help for slash commands, --think / --no-think to control CoT. | | OpenAI server for your apps | rapid-mlx serve qwen3.5-9b-4bit | Point Aider, LibreChat, Open WebUI, or LangChain at http://localhost:8000/v1. | | Agent backends | rapid-mlx serve qwen3.6-35b-8bit &
rapid-mlx agents codex --setup && codex | 10 agents auto-configure via agents --setup once the server is up (12 wire-verified total, 5 Tier-1) — see Agent support. | | Benchmark your Mac | rapid-mlx benchmark run qwen3.5-9b-4bit | Reproducible model-first benchmark; saves privately, with a separate consent-gated benchmark share RUN_ID. |

→ One-shot IDE setup with rapid-mlx launch

---

Agent Support

All 12 agents below are wire-verified against real weights every release via their own integration-test cell. Of these, five are Tier-1 — Claude Code, Codex CLI, Hermes, Aider, and DeepSeek Harness — re-verified end-to-end against the current client binary every release, with one guardian per API wire (Anthropic /v1/messages, OpenAI /v1/responses, and /v1/chat/completions covered for tool-calling depth, reach, and DeepSeek's own harness protocol). The other seven are Tier-2: wire-verified in the matrix and configured on-demand. The first nine agents each ship a rapid-mlx agents --setup config template (except Claude Code, which is one env-var), and Continue.dev gets the same one-command setup via rapid-mlx agents continue --setup (ten setup-capable clients in all, though Continue.dev is not part of the wire-verified matrix below); GitHub Copilot, Factory Droid, and Moonshot Kimi Code plug in through their own documented BYOK config (auth-gated, so the matrix cell is a wire smoke).

Tier-1 is not a label — it is a job that blocks the release. tests/integrations/agent_smoke.sh drives each of the five through the same real multi-step bug-fix task against a local 35B model and asserts the repo's own test suite goes green afterwards; if any one of them fails, the version cannot tag or publish.

Tier-1 (5): Claude Code · Codex CLI · Hermes · Aider — last re-verified end-to-end 2026-07-28 on current binaries (claude 2.1.211, codex 0.145.0, hermes 0.9.0, aider 0.86.2). DeepSeek Harness — promoted 2026-08-17, verified on dsh 0.1.0-rc.7 against qwen3.6-35b-8bit. Tier-2 (7): OpenCode · Qwen Code · OpenHands · Kilo Code · GitHub Copilot · Factory Droid · Moonshot Kimi Code.

| Agents (12) | Frameworks (3) | |---|---| | Codex CLI · Claude Code · OpenCode · Qwen Code · OpenHands · Hermes Agent · Aider · Kilo Code · DeepSeek Harness · GitHub Copilot · Factory Droid · Moonshot Kimi Code | LangChain (+ LangGraph) · PydanticAI · smolagents |

Also compatible with OpenAI-compatible clients that allow direct local endpoints via http://localhost:8000/v1 — LibreChat, Open WebUI, and more plug in with a single URL change.

→ Full 12-agent + 3-framework matrix (test cells + xfail reasons) → Codex CLI · Claude Code · OpenCode · Qwen Code · OpenHands · Hermes · Aider · Kilo Code · DeepSeek Harness · Copilot · Droid · Kimi Code

---

Latest Large-Model Benchmarks

Single-request (B=1) serving on a 256 GB M3 Ultra, with one 4-bit model resident. Each row is the median of three requests after server readiness, with the prefix cache cleared before every run. These are local serving measurements, not model quality scores. All three rows target an 8K prompt and 256-token decode; after chat templating the Qwen prompts contain 8,156 tokens and the GLM prompt contains 8,192.

| Model | Shape | Median TTFT | Prefill | Decode | MLX memory observed through 32K | |---|---|---:|---:|---:|---:| | qwen3.8-27b-4bit | 27B dense | 24.66s | 330.8 tok/s | 43.4 tok/s | 26.7 GB active / 27.1 GB peak | | qwen3.8-flash-next-4bit | 180B total / 6B active¹ | 9.40s | 867.9 tok/s | 23.0 tok/s | 102.8 GB active / 148.1 GB peak | | glm5.3-flash-4bit | 320B total / 18B active | 22.78s | 359.6 tok/s | 27.8 tok/s | 180.6 GB active / 195.6 GB peak |

The biggest 0.13.4 gain is Qwen3.8-27B's verified MTP path. On the same model, Mac, dependency versions, prompts, and cache-clear procedure:

| Prompt | 0.13.3 decode | 0.13.4 decode | Speedup | TTFT (0.13.3 → 0.13.4) | |---:|---:|---:|---:|---:| | 128 | 30.71 tok/s | 43.93 tok/s | 1.43× | 0.54s → 0.51s | | 2K | 28.88 tok/s | 44.69 tok/s | 1.55× | 6.18s → 6.07s | | 8K | 24.58 tok/s | 43.38 tok/s | 1.77× | 25.04s → 24.66s | | 32K | 16.52 tok/s | 38.66 tok/s | 2.34× | 110.25s → 109.12s |

¹ Flash-Next comprises a 125B language model, 51B n-gram embedding, and 4B MTP head. Its default autoregressive path was effectively flat between the two builds; separately qualified MTP and prompt-lookup modes remain workload-dependent.

Flash-Next prefill reaches 262 / 875 / 868 / 716 tok/s at 128 / 2K / 8K / 32K; median TTFT is 0.35 / 2.30 / 9.40 / 45.73s. Its 99 GB quantized weights make 192 GB the practical recommended tier; 128 GB is tight and was not physically tested. GLM-5.3-Flash reached 27.2 tok/s decode at 32K, but its TTFT was 118.34s and its 195.6 GB allocator peak leaves no safe system headroom on a 192 GB Mac. Use a 256 GB Mac for this measured 32K workload; 192 GB remains the catalog floor for shorter contexts. Qwen3.8-27B remains the 32 GB recommendati