FareedKhan-dev/kimi-k3-in-c

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental
--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Slow, and answering correctly, in 8.24 GB, from a checkpoint of 1.56 TB. This particular batch command deliberately asks for a raw continuation. The official Kimi K3 checkpoint is also chat-capable; use the XTML REPL below when you want answers and multi-turn history. Give the same batch request more memory and the answer does not change, only the clock:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
           --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental
--- generated text ---
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci
----------------------
28 tokens in 299.3 s, 10.69 s/token average
PEAK RSS for the whole run: 127.92 GB

Every figure in this document comes from the measurement output in docs/data/.

The dense trunk stays in memory to whatever depth you choose and streams the rest; the 1.45 TB of routed experts are never resident, and are multiplied straight out of their packed 4-bit form. The consequence is that the same model runs in 8 GB and in 224 GB and produces byte-identical output at every budget between.

Four decisions about where bytes live take it from a cluster to a laptop, and the answer at the bottom is the same as the answer at the top:

Part II builds every box in both diagrams from scratch, one component at a time.

---

Contents

Part I: Getting started

• Requirements
• Quick start: clone, build and verify in about a minute, with no model
• Full setup: the whole path to generated text
• Usage
• Synopsis
• Prompt options
• Memory options
• Generation options
• Diagnostic options
• Exit codes
• Environment variables
• Worked examples
• Choosing a preset
• Reading the run report
• Common questions

• The problem: a model that does not fit
• The four reductions
• The machine, and what it assumes
• The codebase
• Three invariants
• 1. Reading a 1.56 TB checkpoint from its headers
• 2. The config reader that refuses to guess
• 3. The tokenizer, byte for byte
• 4. Reduction one: the experts already ship at half a byte
• 5. Kernels with a floating point contract
• 6. Reduction two: KDA, attention with a memory that never grows
• 7. Reduction three: MLA, one latent instead of ninety-six heads
• 8. Attention residuals: layers that look back
• 9. Picking 16 experts of 896
• 10. Packing the trunk: 93 layers, one read each
• 11. Reduction four: streaming the trunk turns a floor into a dial
• 12. An LRU cache for the experts
• 13. How big should that cache be? Ask the trace

• The gate ladder
• A tiny oracle first
• Proving it on the full checkpoint
• The first tokens
• Sustained generation: text in, text out

• The memory ladder: 8 GB to 224 GB
• The cache that was not participating
• Allocation beats capacity
• Measuring the measurement
• Storage is the whole game
• Why the trunk is not quantised

• Scope
• Closing the ledger
• Documentation
• Development
• Star history
• License

Part I: Getting started

Requirements

The gate is storage: the checkpoint is 1.56 TB. Everything else is ordinary.

| | | | |---|---|---| | OS | Linux, x86-64 (reference); macOS/arm64 and Windows/x86-64 also build and pass every gate | uses O_DIRECT, posix_memalign, getrusage -- ported for Windows via MSYS2's MinGW-w64 (see src/io/k3_portable_io.h) | | CPU | AVX2 + FMA on x86-64, NEON on arm64 | AVX-512 unnecessary. make portable targets generic AVX2 on x86-64 | | RAM | 8 GB and up | every preset works; more memory is faster, never different | | Storage | ~1.7 TB free | 1.56 TB checkpoint + 109 GB packed trunk, ideally on fast local disk | | Toolchain | GCC ≥ 9 or Clang ≥ 10 | GNU make, or CMake | | Python | 3.9+ | for the download, pack and analysis tools; not for make test |

The tokenizer and config reader are portable C99 and build anywhere. Without a checkpoint you can still do everything in Quick start.

Quick start

Clone, build and run the entire test suite. No checkpoint, no network, no Python. The whole thing takes about a minute.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j            # seconds. Seven C files, a compiler and OpenMP
make test          # under a minute

It ends like this, or it failed:

GATE 1  teacher forcing : 32/32 positions match tf_pred
        generated span  : 20/20  <- must be exact
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental    : 20/20 generated tokens match full_ids  <- KV cache + carried KDA state
VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY
ALL WEIGHTLESS TESTS PASSED

That is the whole engine: every kernel, the streaming cache, the safetensors reader, the config reader, the tokenizer, and an end-to-end oracle over a 13-layer model built with the same tensor graph as the released one, checked against a PyTorch reference from fixtures committed to the repository.

One published measurement also replays on the spot, from a trace recorded during a full 93-layer run (this one needs Python 3.9+ and numpy):

python3 tools/sim_cache.py tests/fixtures/expert_trace.bin

100,096 expert requests, reprinting the capacity table in expert-cache-capacity.txt.

Full setup

Six steps from an empty directory to generated text. Only step 4 is slow.

./scripts/k3-doctor.sh can be run at any point. It checks the toolchain, sizes your RAM to a preset, measures your storage, and prints the exact command to run next.

Step 0. clone

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

About 45 MB, most of it the diagrams and the test fixtures.

Step 1. check the machine

./scripts/k3-doctor.sh

Takes about a minute, because it measures your disk the way the engine reads it. It exits non-zero if the machine cannot run the model at all.

Step 2. build

make -j

Seconds. The only dependencies are a C99 compiler, libm and OpenMP. CMake works too:

cmake -B build && cmake --build build -j && ctest --test-dir build

Step 3. verify before downloading anything

make test

This is worth doing before committing to a 1.56 TB download: it proves the engine matches its reference on a model with the same tensor graph, and it needs nothing but the repository.

Step 4. fetch the checkpoint

1.56 TB, so hours rather than minutes. Get a token from huggingface.co/settings/tokens:

export HF_TOKEN=hf_your_token_here          # read from the environment, never echoed
./scripts/download-model.sh ~/k3model       # resumable, re-run to continue

The script finishes by verifying the shard count, the exact byte total, and then every individual per-shard size against the published figures:

verifying…
  shards : 96 (expect 96)
  bytes  : 1560936091448 (expect 1560936091448)
  shards : all 96 match their published sizes individually
  RESULT : byte-exact match

A partial download does not fail loudly; it produces wrong tokens. Treat a FAIL here as a stop. Checking per shard also turns "re-download 1.56 terabytes" into "re-download this one 17 gigabyte file", and it catches the one case a total cannot: two shards wrong in opposite directions by the same amount.

Step 5. pack the trunk

./scripts/pack-trunk.sh ~/k3model ~/k3trunk

About four minutes, once. It rewrites the 93 dense layers into one 109 GB file where layer L lives at a known offset and can be read in a single call. This is what turns the memory requirement into a dial. Put the output on the fastest disk you have.

Step 6. run

./bin/k3 ~/k3model --trunk ~/k3trunk --preset workstation \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

The tokenizer ships with the checkpoint, which is why --tok points at the model directory.

Where everything ends up

kimi-k3-in-c/    ~45 MB   source, docs, images, and bin/k3
~/k3model/      1.56 TB   96 shards · config.json · tiktoken.model · tokenizer_config.json
~/k3trunk/       109 GB   trunk.bin · trunk.json, on the fastest disk you have

The first token of any run loads every pinned layer from disk, about 108 GB at the server preset, so it takes far longer than the steady rate. That cost is paid once per run, not once per token.

Usage

Synopsis

k3 <model_dir> [prompt] [memory] [generation] [diagnostics]

<model_dir> is the directory holding the .safetensors shards. It is required for any run, but --help, --version and --list-presets work without it:

./bin/k3 --help
./bin/k3 --version
./bin/k3 --list-presets

Prompt options

Outside --chat, exactly one of these is required. Passing none, or more than one, is a usage error (exit 2).

| flag | argument | | |---|---|---| | --prompt | TEXT | tokenize TEXT and run it. Requires --tok. | | --prompt-file | PATH | tokenize the file's bytes. Requires --tok. Preferred for anything non-ASCII: the shell re-encodes argv, whereas a file is read verbatim | | --ids | 1,2,3 | token ids directly. No tokenizer is loaded at all, so this works on a machine with no tokenizer files. The reproducible channel the tests use |

# text in
./bin/k3 ~/k3model --tok ~/k3model --prompt "The capital of France is" ...
text in, from a file. Use this for CJK, emoji, accents
printf 'La capitale de la France est' > /tmp/p.txt
./bin/k3 ~/k3model --tok ~/k3model --prompt-file /tmp/p.txt ...
ids in, ids out, no tokenizer needed
./bin/k3 ~/k3model --ids 1008,10484,318,15383,387 ...

Memory options

| flag | argument | default | | |---|---|---|---| | --preset | NAME | none | laptop · desktop · workstation · server · max. Sets both budgets below | | --trunk | DIR | off | the packed trunk directory from step 5. This is what enables streaming. Without it the trunk loads fully resident, around 113.5 GB | | --trunk-gb | X | 16 | budget for pinned layers plus the streaming ring | | --cache-gb | X | 64 | budget for the routed-expert LRU cache | | --trunk-ring | N | 2 | streaming ring slots: one layer being computed on, the rest reads in flight. Each extra slot costs one slot of RAM, and the budget still wins if it does not fit | | --threads | N | physical cores | OpenMP threads. On Linux the default is the physical core count rather than OpenMP's one thread per logical CPU, which is slower here on SMT parts. OMP_NUM_THREADS, if set, is respected | | --ultra-low-memory | none | off | stream exact embedding rows and lm_head chunks; full recompute also reuses one recurrent-state slot. Requires --trunk |

The ultra preset selects --ultra-low-memory with a 2.5 GB trunk ring and a 0.31 GB expert cache. It is a proof-of-life path for 8 GB-class machines, not an interactive-speed preset; model precision, Top-K routing and all 93 layers are unchanged.

--preset and the two -gb flags set the same two numbers, so a preset is just a shorthand. Order matters if you mix them: a later flag wins, so --preset server --cache-gb 40 gives you the server trunk budget with a 40 GB cache.

--preset without --trunk does nothing useful. Every preset assumes the trunk is

streamed. Omit --trunk and the engine loads all 113.5 GB resident regardless of the

budget you asked for.

Generation options

| flag | argument | default | | |---|---|---|---| | --gen | N | 8 | tokens to generate. Ceiling 4096; prompts may be up to 32768 tokens. --gen 0 --incremental --save-state PATH runs only the prefill and saves its exact state, to warm a prefix once and resume it many times | | --stop-id | N | off | stop as soon as the model emits id N; repeatable, up to 8. The released checkpoint declares two end ids that disagree (163586 in config.json, 163585 in tokenizer_config.json) and emits both, so pass both | | --incremental | none | off | carry the KV cache and the recurrent state between tokens instead of re-running the whole prefix | | --tok | DIR | none | directory holding tiktoken.model and tokenizer_config.json |

Pass --incremental for any generation of length. Without it every step re-runs the entire prefix, which is O(T²); with it, step 0 pays for the prompt and every later step costs the same fixed amount. Both paths are gated on producing identical tokens, so this is a pure speed choice.

Text chat (official Kimi K3 XTML)

K3 is chat-capable. --chat uses the official XTML segments and tokenizer control tokens; it does not fall back to ChatML or a handwritten generic prompt. Version one is text-only: there are no tools, images, server, or context compaction.

./bin/k3 ~/k3model --trunk ~/k3trunk --preset desktop --tok ~/k3model \
  --chat --system "You are a helpful assistant." \
  --history my-session.jsonl --incremental

The REPL accepts ordinary text plus /help, /reset, and /exit. It prints the complete canonical record, including … and …. When --history is supplied, it writes a portable, human-readable JSONL file such as:

{"role":"system","content":"You are a helpful assistant."}
{"role":"user","content":"Explain cache locality."}
{"role":"assistant","content":"…","reasoning_content":"…"}

Treat that file as sensitive: it can contain every user message and the complete assistant reasoning record, which K3 requires to continue a conversation faithfully. On restart the engine validates, re-renders, and re-prefills the transcript; it deliberately does not serialize opaque KV, MLA, or KDA state. Within one session under --incremental the state a turn built is kept, and the next turn prefills only its new tail when the rendered transcript begins with exactly the ids that state was fed (the REPL prints how many positions were reused); /reset or any other divergence starts over. A supplied --system must exactly match an existing initial system record. Literal control-marker text in user messages is encoded as ordinary text, never as an XTML control token.

Chat defaults to --gen 4096 and, like batch generation, to greedy decoding. Passing any of --temperature, --top-p or --seed turns sampling on (temperature 1.0, top-p 0.95 unless given); --greedy forces argmax even then. Sampling uses PCG32; --seed N deterministically derives one stream per assistant