cactus-compute/cactus

Cactus

[![Docs][docs-shield]][docs-url] [![Website][website-shield]][website-url] [![GitHub][github-shield]][github-url] [![HuggingFace][hf-shield]][hf-url] [![Reddit][reddit-shield]][reddit-url] [![Blog][blog-shield]][blog-url]

A hybrid edge-cloud AI engine for mobile devices & wearables.

┌─────────────────┐
│  Cactus Engine  │ ←── OpenAI-compatible APIs for text, speech, and vision.
└─────────────────┘     
         │
┌─────────────────┐
│  Cactus Graph   │ ←── Zero-copy computation graph
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Kernels  │ ←── CPU/GPU kernels for (Apple, Samsung, Pixel, etc.)
└─────────────────┘     
         │
┌─────────────────┐
│ Cactus Quants   │ ←── Custom rotation-based quantization technique
└─────────────────┘  

Quick Demo (Mac)

• Step 1: brew install cactus-compute/cactus/cactus
• Step 2: cactus run

Cactus Engine

#include "cactus_engine.h"
cactus_model_t model = cactus_init(
    "path/to/weight/folder",
    "path to txt or dir of txts for auto-rag",
    false
);
const char* messages = R"([
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "My name is Henry Ndubuaku"}
])";
const char* options = R"({
    "max_tokens": 50,
    "stop_sequences": ["<|im_end|>"]
})";
char response[4096];
int result = cactus_complete(
    model,            // model handle
    messages,         // JSON chat messages
    response,         // response buffer
    sizeof(response), // buffer size
    options,          // generation options
    nullptr,          // tools JSON
    nullptr,          // streaming callback
    nullptr,          // user data
    nullptr,          // pcm audio buffer
    0                 // pcm buffer size
);

{
    "success": true,        // generation succeeded
    "error": null,          // error details if failed
    "cloud_handoff": false, // true if cloud model used
    "response": "Hi there!",
    "function_calls": [],   // parsed tool calls
    "segments": [],         // transcription segments (empty for chat)
    "confidence": 0.8193,   // model confidence
    "confidence_threshold": 0.7, // resolved handoff threshold (model-dependent)
    "time_to_first_token_ms": 45.23,
    "total_time_ms": 163.67,
    "prefill_tps": 1621.89,
    "decode_tps": 168.42,
    "ram_usage_mb": 245.67,
    "prefill_tokens": 28,
    "decode_tokens": 50,
    "total_tokens": 78
}

Cactus Graph

#include "cactus_graph.h"
CactusGraph graph;
auto a = graph.input({2, 3}, Precision::FP16);
auto b = graph.input({3, 4}, Precision::INT8);
auto x1 = graph.matmul(a, b, false);
auto x2 = graph.transpose(x1);
auto result = graph.matmul(b, x2, true);
float a_data[6] = {1.1f, 2.3f, 3.4f, 4.2f, 5.7f, 6.8f};
float b_data[12] = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12};
graph.set_input(a, a_data, Precision::FP16);
graph.set_input(b, b_data, Precision::INT8);
graph.execute();
void* output_data = graph.get_output(result);
graph.hard_reset(); 

Inference Speed

• LLM: Gemma-4-E2B-CQ4 (1k-context prefill / decode for 100 tokens)
• VLM: Gemma-4-E2B-CQ4 (256px image encode time / decode)
• Transcribe: Parakeet-TDT-0.6B-CQ4 (20s audio end-to-end transcribe time)
• 1k-Context RAM: peak MB during the LLM benchmark
• No speculative decode or MTP, pure decode

| Device | LLM | VLM | Transcribe | RAM | |--------|-----|-----|------------|---------------| | Mac M5 Max | 2964tps / 154tps | 0.09s / 168tps | 0.15s | 1348MB | | Mac M4 Pro | 1963tps / 101tps | 0.25s / 112tps | 0.21s | 1225MB | | Mac M3 Pro | 1294tps / 64tps | 0.40s / 72tps | 0.37s | 735MB | | iPad/Vision Pro M5 | 1336tps / 71tps | 0.25s / 80tps | 0.27s | 703MB | | iPhone 17 Pro | 729tps / 37tps | 0.5s / 39tps | 0.51s | 644MB | | iPhone 15 Pro | 517tps / 26tps | 1.15s / 27tps | 0.82s | 633MB |

N/B: With 1k-context prefill and decode for 100 runs on M5 Max

• LFM2.5-VL-1.6B = 289toks/sec
• Qwen3-1.7B = 155toks/sec
• LFM2.5-VL-450m = 472toks/sec, image encodes in 43ms
• LFM22.5-VL-230m = 555toks/sec

Output Quality

• Gemma-4-E2B-it accuracy across bit widths, averaged over 3 seeds.
• CQ3.26 and CQ2.54 are mixed-precision, CQ2/CQ3/CQ4 are uniformly quantized.
• Full results in docs/cactus_quants.md:

Supported Models

• Any HuggingFace model can be converted using cactus convert [HF-Name], though experimental.
• Liquid, Gemma. whisper. parakeet and Qwen model families are especially tested.
• Some models have been pre-uploaded here, just run cactus download [HF-Name].
• cactus run [HF-Name] albeit first downloads or convert the model if not found.

Needle

Needle is a 26m parameter model for on-device tool calling:

cactus run Cactus-Compute/needle [--tools my_tools.json]  # OpenAI function-calling format; demo toolset by default

Learn More

| Reference | Language | Description | |-----------|----------|-------------| | Cactus Engine | C | Chat completion, streaming, tool calling, transcription, embeddings, RAG, vision, vector index, cloud handoff | | Cactus Graph | C++ | Tensor operations, matrix multiplication, attention, normalization, activation functions | | Cactus Kernels | C++ | ARM NEON SIMD kernels for matmul, attention, convolution, quantization, DSP, image processing | | Cactus Quants | C++ | Rotation-and-codebook quantization from 4-bit to 1-bit for all weight tensors | | Cactus Hybrid | C/Python | Route hard queries to the cloud automatically based on local model confidence | | Python Package | Python | Python package and CLI |

Bindings

• Swift
• Kotlin
• Flutter
• React Native
• Python
• Rust

Using this repo

┌────────────────────────────────────────────────────────────────────────────────┐
│                                                                                │
│ Step 0: if on Linux (Ubuntu/Debian)                                            │
│ sudo apt-get install python3.12 python3.12-venv python3-pip cmake              │
│   build-essential libcurl4-openssl-dev                                         │
│                                                                                │
│ Step 1: clone and setup                                                        │
│ git clone https://github.com/cactus-compute/cactus && cd cactus                │
│ source ./setup                                                                 │
│                                                                                │
│ Step 2: use the commands                                                       │
│────────────────────────────────────────────────────────────────────────────────│
│                                                                                │
│  cactus auth                         manage cloud API key                      │
│    --status                          show key status                           │
│    --clear                           remove saved key                          │
│                                                                                │
│  cactus run [model|path]             run a model (downloads if needed)         │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --image                     image file for VLM inference              │
│    --audio                     audio file for audio chat                 │
│    --system                  system prompt                             │
│    --prompt                    send prompt immediately                   │
│    --tools                tool definitions for tool calling         │
│    --thinking                        enable thinking/reasoning mode            │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│                                                                                │
│  cactus transcribe [model]           live microphone transcription with a model│
│    --file                 audio file to transcribe (WAV)            │
│    --language                  language code (default: en)               │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│                                                                                │
│  cactus download [model]             get a bundle (prebuilt, else build)       │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       refresh cached bundle                     │
│                                                                                │
│  cactus convert  [dir]        HuggingFace -> Cactus CQ weights          │
│    --bits 1|2|3|4                    CQ quantization (default: 4)              │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force weight conversion from source       │
│    --lora                      merge a LoRA adapter before converting    │
│    --weights-only                    stop after CQ weights (skip the graph)    │
│    --artifact-dir              bundle output (default: weights/)  │
│                                                                                │
│  cactus serve [model]                OpenAI-compatible local HTTP server       │
│    --host                      bind address (default: 127.0.0.1)         │
│    --port                      port (default: 8080)                      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│    --no-cloud-handoff                disable automatic cloud handoff           │
│    --confidence-threshold <0..1>     handoff to cloud below this confidence    │
│    --cloud-timeout-ms             max wait for cloud handoff                │
│                                                                                │
│  cactus code                         run the AI coding agent (TUI / print)     │
│    --serve-model                 auto-start a server with this model       │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild from source           │
│    --host                      server address (default: 127.0.0.1)       │
│    --port                      server port (default: 8080)               │
│    --no-serve                        require a running server (no auto-start)  │
│    --no-cloud-handoff                disable automatic cloud handoff           │
│    --confidence-threshold <0..1>     handoff to cloud below this confidence    │
│    --cloud-timeout-ms             max wait for cloud handoff                │
│    --                       pass remaining args to the agent          │
│                                                                                │
│  cactus list                         list downloaded models                    │
│                                                                                │
│  cactus build                        build cactus libraries                    │
│    --apple                           Apple (iOS/macOS)                         │
│    --android                         Android                                   │
│    --python                          shared lib for Python FFI                 │
│                                                                                │
│  cactus test                         run the test suite                        │
│    --component                 kernels | graph | engine | all            │
│                                      (default: all)                            │
│    --model                    default: google/gemma-4-E2B-it            │
│    --transcription-model      default: nvidia/parakeet-tdt-0.6b-v3      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --token                    HuggingFace token (gated models)          │
│    --reconvert                       force local rebuild of test models        │
│    --suite                     run a single test suite by name           │
│                                      (resolved across components,              │
│                                      e.g. llm → engine)                        │
│    --list                            list components and suites                │
│    --ios                             run on connected iPhone                   │
│    --android                         run on connected Android                  │
│    --enable-telemetry                send cloud telemetry (off by default)     │
│                                                                                │
│  cactus benchmark                    run the engine benchmark suite            │
│    --model                    default: google/gemma-4-E2B-it            │
│    --transcription-model      default: nvidia/parakeet-tdt-0.6b-v3      │
│    --bits 1|2|3|4|2.54|3.26          CQ quantization (default: 4)              │
│    --backend cpu|metal               inference backend (default: auto)         │
│    --ios                             run on connected iPhone                   │
│    --android                         run on connected Android                  │
│                                                                                │
│  cactus clean                        delete build artifacts, weights, venv     │
│  cactus --help                       show this help                            │
│                                                                                │
└────────────────────────────────────────────────────────────────────────────────┘

@software{cactus,
  title        = {Cactus: AI Inference Engine for Phones & Wearables},
  author       = {Ndubuaku, Henry and Cactus Team},
  url          = {https://github.com/cactus-compute/cactus},
  year         = {2025}
}