FareedKhan-dev/kimi-k3-in-c

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
           --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental
--- generated text ---
 Paris.",
+            "The Eiffel
----------------------
8 tokens in 261.5 s, 32.69 s/token average
PEAK RSS for the whole run: 8.24 GB

Slow, and answering correctly, in 8.24 GB, from a checkpoint of 1.56 TB. This is a base model, so what follows " Paris." is a continuation rather than a reply; there is no chat template. Give it more memory and the answer does not change, only the clock:

$ ./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
           --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental
--- generated text ---
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci
----------------------
28 tokens in 299.3 s, 10.69 s/token average
PEAK RSS for the whole run: 127.92 GB

Every figure in this document comes from the measurement output in docs/data/.

The dense trunk stays in memory to whatever depth you choose and streams the rest; the 1.45 TB of routed experts are never resident, and are multiplied straight out of their packed 4-bit form. The consequence is that the same model runs in 8 GB and in 224 GB and produces byte-identical output at every budget between.

Four decisions about where bytes live take it from a cluster to a laptop, and the answer at the bottom is the same as the answer at the top:

Part II builds every box in both diagrams from scratch, one component at a time.

---

Contents

Part I: Getting started

• Requirements
• Quick start: clone, build and verify in about a minute, with no model
• Full setup: the whole path to generated text
• Usage
• Synopsis
• Prompt options
• Memory options
• Generation options
• Diagnostic options
• Exit codes
• Environment variables
• Worked examples
• Choosing a preset
• Reading the run report
• Common questions

Part II: How it works

• The problem: a model that does not fit
• The four reductions
• The machine, and what it assumes
• The codebase
• Three invariants
• 1. Reading a 1.56 TB checkpoint from its headers
• 2. The config reader that refuses to guess
• 3. The tokenizer, byte for byte
• 4. Reduction one: the experts already ship at half a byte
• 5. Kernels with a floating point contract
• 6. Reduction two: KDA, attention with a memory that never grows
• 7. Reduction three: MLA, one latent instead of ninety-six heads
• 8. Attention residuals: layers that look back
• 9. Picking 16 experts of 896
• 10. Packing the trunk: 93 layers, one read each
• 11. Reduction four: streaming the trunk turns a floor into a dial
• 12. An LRU cache for the experts
• 13. How big should that cache be? Ask the trace

Part III: Validation

• The gate ladder
• A tiny oracle first
• Proving it on the full checkpoint
• The first tokens
• Sustained generation: text in, text out

Part IV: Measurements

• The memory ladder: 8 GB to 224 GB
• The cache that was not participating
• Allocation beats capacity
• Measuring the measurement
• Storage is the whole game
• Why the trunk is not quantised

Part V: Reference

• Scope
• Closing the ledger
• Documentation
• Development
• License

---

Part I: Getting started

Requirements

The gate is storage: the checkpoint is 1.56 TB. Everything else is ordinary.

| | | | |---|---|---| | OS | Linux, x86-64 (reference); macOS/arm64 and Windows/x86-64 also build and pass every gate | uses O_DIRECT, posix_memalign, getrusage -- ported for Windows via MSYS2's MinGW-w64 (see src/io/k3_portable_io.h) | | CPU | AVX2 + FMA | AVX-512 unnecessary. make portable targets generic AVX2 | | RAM | 8 GB and up | every preset works; more memory is faster, never different | | Storage | ~1.7 TB free | 1.56 TB checkpoint + 109 GB packed trunk, ideally on fast local disk | | Toolchain | GCC ≥ 9 or Clang ≥ 10 | GNU make, or CMake | | Python | 3.9+ | for the download, pack and analysis tools; not for make test |

The tokenizer and config reader are portable C99 and build anywhere. Without a checkpoint you can still do everything in Quick start.

Quick start

Clone, build and run the entire test suite. No checkpoint, no network, no Python. The whole thing takes about a minute.

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c
make -j            # seconds. Seven C files, a compiler and OpenMP
make test          # under a minute

It ends like this, or it failed:

GATE 1  teacher forcing : 32/32 positions match tf_pred
        generated span  : 20/20  <- must be exact
GATE 2  greedy decode   : 20/20 generated tokens match full_ids
GATE 3  incremental    : 20/20 generated tokens match full_ids  <- KV cache + carried KDA state
VERDICT: ENGINE MATCHES THE REFERENCE EXACTLY
ALL WEIGHTLESS TESTS PASSED

That is the whole engine: every kernel, the streaming cache, the safetensors reader, the config reader, the tokenizer, and an end-to-end oracle over a 13-layer model built with the same tensor graph as the released one, checked against a PyTorch reference from fixtures committed to the repository.

One published measurement also replays on the spot, from a trace recorded during a full 93-layer run (this one needs Python 3.9+ and numpy):

python3 tools/sim_cache.py tests/fixtures/expert_trace.bin

100,096 expert requests, reprinting the capacity table in expert-cache-capacity.txt.

Full setup

Six steps from an empty directory to generated text. Only step 4 is slow.

./scripts/k3-doctor.sh can be run at any point. It checks the toolchain, sizes your RAM to a preset, measures your storage, and prints the exact command to run next.

Step 0. clone

git clone https://github.com/FareedKhan-dev/kimi-k3-in-c.git
cd kimi-k3-in-c

About 45 MB, most of it the diagrams and the test fixtures.

Step 1. check the machine

./scripts/k3-doctor.sh

Takes about a minute, because it measures your disk the way the engine reads it. It exits non-zero if the machine cannot run the model at all.

Step 2. build

make -j

Seconds. The only dependencies are a C99 compiler, libm and OpenMP. CMake works too:

cmake -B build && cmake --build build -j && ctest --test-dir build

Step 3. verify before downloading anything

make test

This is worth doing before committing to a 1.56 TB download: it proves the engine matches its reference on a model with the same tensor graph, and it needs nothing but the repository.

Step 4. fetch the checkpoint

1.56 TB, so hours rather than minutes. Get a token from huggingface.co/settings/tokens:

export HF_TOKEN=hf_your_token_here          # read from the environment, never echoed
./scripts/download-model.sh ~/k3model       # resumable, re-run to continue

The script finishes by verifying the shard count, the exact byte total, and then every individual per-shard size against the published figures:

verifying…
  shards : 96 (expect 96)
  bytes  : 1560936091448 (expect 1560936091448)
  shards : all 96 match their published sizes individually
  RESULT : byte-exact match

A partial download does not fail loudly; it produces wrong tokens. Treat a FAIL here as a stop. Checking per shard also turns "re-download 1.56 terabytes" into "re-download this one 17 gigabyte file", and it catches the one case a total cannot: two shards wrong in opposite directions by the same amount.

Step 5. pack the trunk

./scripts/pack-trunk.sh ~/k3model ~/k3trunk

About four minutes, once. It rewrites the 93 dense layers into one 109 GB file where layer L lives at a known offset and can be read in a single call. This is what turns the memory requirement into a dial. Put the output on the fastest disk you have.

Step 6. run

./bin/k3 ~/k3model --trunk ~/k3trunk --preset workstation \
         --tok ~/k3model --prompt "The capital of France is" --gen 8 --incremental

The tokenizer ships with the checkpoint, which is why --tok points at the model directory.

Where everything ends up

kimi-k3-in-c/    ~45 MB   source, docs, images, and bin/k3
~/k3model/      1.56 TB   96 shards · config.json · tiktoken.model · tokenizer_config.json
~/k3trunk/       109 GB   trunk.bin · trunk.json, on the fastest disk you have

The first token of any run loads every pinned layer from disk, about 108 GB at the server preset, so it takes far longer than the steady rate. That cost is paid once per run, not once per token.

Usage

Synopsis

k3 <model_dir> [prompt] [memory] [generation] [diagnostics]

<model_dir> is the directory holding the .safetensors shards. It is required for any run, but --help, --version and --list-presets work without it:

./bin/k3 --help
./bin/k3 --version
./bin/k3 --list-presets

Prompt options

Exactly one of these is required. Passing none, or more than one, is a usage error (exit 2).

| flag | argument | | |---|---|---| | --prompt | TEXT | tokenize TEXT and run it. Requires --tok. | | --prompt-file | PATH | tokenize the file's bytes. Requires --tok. Preferred for anything non-ASCII: the shell re-encodes argv, whereas a file is read verbatim | | --ids | 1,2,3 | token ids directly. No tokenizer is loaded at all, so this works on a machine with no tokenizer files. The reproducible channel the tests use |

# text in
./bin/k3 ~/k3model --tok ~/k3model --prompt "The capital of France is" ...
text in, from a file. Use this for CJK, emoji, accents
printf 'La capitale de la France est' > /tmp/p.txt
./bin/k3 ~/k3model --tok ~/k3model --prompt-file /tmp/p.txt ...
ids in, ids out, no tokenizer needed
./bin/k3 ~/k3model --ids 1008,10484,318,15383,387 ...

Memory options

| flag | argument | default | | |---|---|---|---| | --preset | NAME | none | laptop · desktop · workstation · server · max. Sets both budgets below | | --trunk | DIR | off | the packed trunk directory from step 5. This is what enables streaming. Without it the trunk loads fully resident, around 113.5 GB | | --trunk-gb | X | 16 | budget for pinned layers plus the streaming ring | | --cache-gb | X | 64 | budget for the routed-expert LRU cache | | --ultra-low-memory | none | off | stream exact embedding rows and lm_head chunks; full recompute also reuses one recurrent-state slot. Requires --trunk |

The ultra preset selects --ultra-low-memory with a 2.5 GB trunk ring and a 0.31 GB expert cache. It is a proof-of-life path for 8 GB-class machines, not an interactive-speed preset; model precision, Top-K routing and all 93 layers are unchanged.

--preset and the two -gb flags set the same two numbers, so a preset is just a shorthand. Order matters if you mix them: a later flag wins, so --preset server --cache-gb 40 gives you the server trunk budget with a 40 GB cache.

--preset without --trunk does nothing useful. Every preset assumes the trunk is

streamed. Omit --trunk and the engine loads all 113.5 GB resident regardless of the

budget you asked for.

Generation options

| flag | argument | default | | |---|---|---|---| | --gen | N | 8 | tokens to generate. Ceiling 4096; prompts may be up to 32768 tokens | | --incremental | none | off | carry the KV cache and the recurrent state between tokens instead of re-running the whole prefix | | --tok | DIR | none | directory holding tiktoken.model and tokenizer_config.json |

Pass --incremental for any generation of length. Without it every step re-runs the entire prefix, which is O(T²); with it, step 0 pays for the prompt and every later step costs the same fixed amount. Both paths are gated on producing identical tokens, so this is a pure speed choice.

Diagnostic options

| flag | argument | | |---|---|---| | --config | PATH | model config; defaults to <model_dir>/config.json | | --layers | N | bind only the first N layers, for partial shard sets | | --out | FILE | JSON results (default k3_run.json) | | --dump-logits | PATH | float32 logits for the first step, for elementwise comparison | | --dump-cache-trace | DIR | writes expert_hist.json and expert_trace.bin, which tools/sim_cache.py replays |

Exit codes

Scripts can rely on these.

| | | |---:|---| | 0 | success | | 1 | a tensor failed to bind, or a forward pass failed | | 2 | usage error, or a config that could not be read with confidence; the engine declines to guess | | 4 | the run finished, but at least one routed expert failed to load, so the emitted ids are unsound. Distinct from 1 because the process otherwise succeeded, and it is the code that catches silent numerical corruption |

Environment variables

| variable | used by | | |---|---|---| | HF_TOKEN | download-model.sh | HuggingFace token, read from the environment and never echoed | | OMP_NUM_THREADS | the engine | thread count, defaulting to all cores | | K3_TOK_FILES | tokenizer tools and CI | directory holding tiktoken.model, when it is not in a default location | | K3_MODEL_DIR | tools/budget.py | checkpoint directory, when not given as an argument |

Worked examples

# Full-model, one-token proof of life on an 8 GB-class ARM64 machine.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset ultra \
         --tok ~/k3model --prompt "The capital of France is" --gen 1
Smallest possible run, the 8 GB floor.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset laptop \
         --tok ~/k3model --prompt "Hello! My name is" --gen 16 --incremental
Fastest per gigabyte. Pins 90 of 93 trunk layers.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
         --tok ~/k3model --prompt "def fibonacci(n):" --gen 28 --incremental
Hand-tuned split instead of a preset: everything to the trunk.
./bin/k3 ~/k3model --trunk ~/k3trunk --trunk-gb 110 --cache-gb 13 \
         --tok ~/k3model --prompt-file prompt.txt --gen 32 --incremental
Reproducible: ids in, ids out, no tokenizer, JSON results.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset desktop \
         --ids 1008,10484,318,15383,387 --gen 8 --incremental --out run.json
Capture a cache trace, then replay it offline at any capacity.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset workstation \
         --ids 1008,10484,318,15383,387 --gen 8 --incremental \
         --dump-cache-trace /tmp/trace
python3 tools/sim_cache.py /tmp/trace/expert_trace.bin
Elementwise logit comparison against the PyTorch reference.
./bin/k3 ~/k3model --trunk ~/k3trunk --preset server \
         --ids 3,4,5,6,7 --gen 1 --dump-logits /tmp/c_logits.bin
python3 tools/cmp_logits.py /tmp/c_logits.bin ref_logits.json
Partial shard set: bind only the first 8 layers.
./bin/k3 ~/k3model --trunk ~/k3trunk --layers 8 \
         --ids 1,2,3 --gen 1
Under a hard memory ceiling, which is how the ladder was measured.
systemd-run --scope --user -q -p MemoryMax=8G -p MemorySwapMax=0 \
  ./bin/k3 ~/k3model --trunk ~/k3trunk --trunk-gb 2.5 --cache-gb 0.5 \
           --ids 1008,10484,318,15383,387 --gen 8 --incremental

Choosing a preset

$ ./bin/k3 --list-presets
presets (trunk / expert-cache, in GB):
  ultra          2.50 / 0.31    ~3 GB planned: streamed model tables, one state slot. Slow.
  laptop         3.00 / 1.00    8.2 GB peak RSS. The ordinary-path floor.
  desktop       16.00 / 10.00   31.9 GB peak RSS.
  workstation   60.00 / 30.00   95.5 GB peak RSS; the expert cache starts to matter here.
  server       110.00 / 13.00   ~128 GB peak RSS; 90 of 93 trunk layers pinned. Fastest.
  max          110.00 / 109.00  ~224 GB peak RSS; trunk pinned and a large expert cache.
All presets stream the trunk, so they need --trunk <packed_dir>.
Run scripts/k3-doctor.sh to see which one this machine fits.

The boundaries come from the measured ladder, and the doctor keys on MemAvailable rather than MemTotal:

if   [ "$AVAIL_GB" -ge 192 ]; then PRESET=server;      EXPECT="~6 s/token"
elif [ "$AVAIL_GB" -ge  96 ]; then PRESET=workstation; EXPECT="~6-20 s/token"
elif [ "$AVAIL_GB" -ge  32 ]; then PRESET=desktop;     EXPECT="~24 s/token"
elif [ "$AVAIL_GB" -ge  10 ]; then PRESET=laptop;      EXPECT="~27 s/token"
else PRESET=""; fi

Two things worth knowing before you pick:

• max is not faster than server in these measurements. The extra 96 GB buys nothing

outside the noise floor.

• Give the trunk memory before the expert cache. At a fixed 128 GB budget that was

worth 1.69×. Allocation beats capacity has the data.

Reading the run report

The engine prints a memory plan, then a line per generated token, then a summary. Abridged from a workstation run:

cache [final step]
  requests     : 1472  hits 1472 (100.00%)  misses 0  evictions 729
                 TRUE resident hit rate 50.48%
I/O share of wall clock: 71.1%  (trunk 62.4 s + experts 34.2 s of 135.8 s)
trunk [final]
  pinned 48/93 layers, ring 1 slots
  read 368.65 GB in 62.40 s (5908 MB/s)
PEAK RSS for the whole run: 94.74 GB   <- quote this, not the plan

Three numbers carry the meaning:

• TRUE resident hit rate: experts served from RAM. The raw hits counter also

counts experts the prefetcher pulled off disk moments earlier, so it reads 100% at every cache size; the resident figure is printed beneath it.

• I/O share of wall clock: whole-run disk time against total, measured between 41%

and 61% across the ladder.

• PEAK RSS: from getrusage, after the run. This is the memory figure; the up-front

plan runs slightly above it.

Common questions

Memory sits near 113 GB even at a small preset. --trunk was omitted. Without a packed trunk directory the whole trunk is loaded resident; every preset assumes streaming.

A non-ASCII prompt tokenizes oddly. The shell re-encodes argv, so the engine receives different bytes than you typed. Put the prompt in a file and use --prompt-file, which is read verbatim.

--prompt/--prompt-file need --tok DIR. The tokenizer ships with the checkpoint, so add --tok ~/k3model. The engine exits rather than guessing where the vocabulary lives. To skip the tokenizer entirely, pass token ids with --ids.

Throughput is well below the table. Almost always storage. `python3 tools/devbw.py ` measures the disk the way the engine r