chenglou/handwired

Handwired

Turn your regular JavaScript code into a neural net; no training, exact, yet also fully differentiable!

How It Works

A program's made of if-else, loops, and state. Handwired transforms them into matrix multiplication + reglu over floats. For example:

const a = 5
const result = (a > 4) ? 10 : 12

The above can be expressed as basic arithmetic:

const a = 5
// Math.max(0, x) is ReLU. For an integer a, above4 is 1 if a > 4, and 0 otherwise
const above4 = Math.max(0, a - 4) - Math.max(0, a - 5)
// Both branches compute, and above4 picks one
const result = above4  10 + (1 - above4)  12

You can calculate that by hand and see that both arrive at the same result.

We then collect the + and * and relu into a giant matmul, and either run it on JS or WebGPU, for fun.

Because we basically only use these 2 primitives:

• the full program stays differentiable
• the weights don't need training since they're written manually
• existing gpu techniques, such as fusion, sparsity, reordering, have a great time optimizing such minimal primitives

• unfreezing the weights and train them (what does that even do? Let's find out!)
• attaching a specific frozen program to a generic neural net and train end-to-end (now a nanoGPT can do both Shakespeare and your particular UI?)
• achieving GPU-level of parallelism with JS code that was written in a traditional single-threaded way (see the JSON search demo that runs in milliseconds over 100k documents)

Play With It

Try the demos live: https://chenglou.me/handwired/

git clone && bun install, then bun start to run them locally. Check docs/, src/ and template/ to get started! (lab/ is a bunch of old experiments you can check or ignore)