harvard-edge/cs249r_book

Machine Learning Systems

English • 中文 • 日本語 • 한국어
_{The translations follow an earlier version of this page. Help bringing them up to date is welcome.}

---

Why This Curriculum Exists

AI engineering is the discipline of building efficient, reliable, and safe intelligent systems that operate in the real world, not models in isolation. It deserves a place alongside software engineering and computer engineering, and this curriculum teaches it the way computer architecture has long been taught, from first principles and with numbers a student can recompute.

Those numbers come from constraints. A model that scores well on a benchmark still has to fit in memory, meet a latency budget, stay within a power limit, and fail safely in front of real people. These constraints decide the architecture, and they follow laws that outlast any framework or chip. Memory bandwidth, network latency, energy, and failure rates govern a microcontroller with 256 KB of memory and an accelerator in a datacenter alike, even though the two differ in scale by many orders of magnitude. I learned this while working on TinyML at Google, and it became the idea that runs through every volume: constraints drive architecture.

Teaching AI engineering this way takes more than a book, which is why everything lives in one repository. Each part teaches something the others cannot. Students who only read do not internalize, and students who only code do not generalize. The textbook builds the mental model, TinyTorch makes it concrete by having students build the machinery themselves, and the hardware kits and the MLSys·im simulator show where the model breaks, on devices students can hold and at scales beyond any one lab.

The work began in a classroom. The HarvardX TinyML course came first and reached more than 100,000 learners. In 2023, the students in CS249r at Harvard wrote the first class notes for this curriculum together, one topic per group, and we published them on GitHub. Since then, I have kept revising the material with help from a growing community of contributors, and instructors at universities around the world now teach from it. The goal is to reach one million learners by 2030, because the world is building AI factories faster than it is training AI engineers.

---

What You Will Learn

Each chapter connects an idea a reader may already know from machine learning to the systems that make it work in practice.

You know...		You will learn...
How to train a model	→	How training scales across accelerator clusters
That quantization shrinks models	→	How low-precision arithmetic maps to silicon
What a transformer is	→	Why the KV cache dominates memory at inference
Models run on accelerators	→	How serving systems batch requests to balance latency against throughput
Models need good data	→	How data pipelines, selection, and drift decide what a model can learn
Models get deployed	→	How to benchmark, monitor, and roll back models in production
Hardware fails sometimes	→	How to checkpoint and recover training jobs that span thousands of accelerators
Edge devices have limits	→	How to co-design models and hardware
AI uses a lot of energy	→	How to measure and reduce the energy and carbon cost of a system over its full lifecycle
How to prompt an LLM	→	How to architect agent loops, tool sandboxes, and context memory (Vol III, in development)
How vision and robotics models infer	→	How to close real-time sensor-actuator loops under physical constraints (Vol IV, in development)

---

The Four Volumes

The textbook spans four volumes. Each one widens the boundary of the system and raises the cost of failure, so the volumes differ in scope, not in depth.

	Volume	Unit of work	Core systems question	Status
📗	I. Introduction to Machine Learning Systems	The model	How does intelligence execute efficiently on a single machine?	Released; forthcoming in print from MIT Press
📘	II. Scaling Machine Learning Systems	The fleet	How does intelligence scale across distributed clusters and datacenters?	Preview; in active revision
🟣	III. Agentic Machine Learning Systems	The trajectory	How is intelligence governed when it acts autonomously over long horizons?	In development; changes often
🌲	IV. Physical AI: Machine Learning Systems That Sense and Act	The physical action	How can intelligence act safely on matter and physical systems?	In development; changes often

Each volume answers a question the previous one leaves open.

• Vol I → Vol II, when one machine is not enough. Volume I covers single-machine execution, memory walls, and kernel efficiency. Volume II carries those foundations across thousands of accelerators, collective communication, fault-tolerant networks, and datacenter orchestration.
• Vol II → Vol III, when a stateless response is not enough. Volume II serves stateless requests at scale. Volume III adds stateful, multi-step loops that manage context memory, call tools, run in sandboxes, and recover from nondeterministic failures.
• Vol III → Vol IV, when software acts on the physical world. Volume III governs software environments. Volume IV crosses into physical systems, where computational latency becomes distance traveled and a failure cannot be undone.

I am writing Volumes III and IV because I plan to teach both subjects, and because they are where machine learning systems are heading. The progression is a natural one. Volumes I and II run in an open loop, where a request goes in and a response comes out. Volume III closes the loop in software, where an agent acts, observes the result, and acts again. Volume IV closes it through the physical world. Writing a book is how I learn a subject well enough to teach it.

⚠️ Volumes III and IV are open drafts. Chapters are added, reorganized, and rewritten as my understanding grows, so please do not cite or teach from them yet. Feedback is welcome through the book feedback issue forms.

---

What's in the Repository

Around the textbook sit the companion tools that let students build, measure, and practice what they read. The pieces are designed to work together, but none of them requires the others.

	Component	What it does	Status	Links
📖	Textbook	Four volumes that build the mental models, from one machine to fleets, agents, and physical systems.	Vol I released	Read online
🔥	Tiny🔥Torch	Build an ML framework from scratch in 20 progressive modules, from tensors to a transformer.	Released	Get started · Repo guide
🔮	MLSys·im	Model memory, compute, network, and energy limits for systems too large to run in a classroom. The textbook's calculations are built on it.	Released	Use simulator · Repo guide
🛠️	Hardware Kits	Deploy models to Arduino, Seeed, Grove, and Raspberry Pi devices, with real memory limits, power budgets, and latency.	Released	Browse kits · Repo guide
🔬	Labs	Interactive Marimo notebooks, run in the browser, that pair with textbook chapters. A student predicts an outcome, changes a parameter, and sees what breaks.	Early release, built in the open	Try labs · Repo guide
💼	StaffML	More than 9,000 systems design questions across cloud, edge, mobile, and TinyML, with practice drills and mock interviews.	Early release	Practice · Repo guide
🎓	Instructor Hub	A syllabus for each course, with weekly readings and slides: introductory, scaling, TinyML, Edge AI, Physical AI (draft), and Agentic AI (in preparation).	Released	View hub · Repo guide
🎬	Lecture Slides	Beamer decks for each chapter of all four volumes, in one shared theme that is easy to restyle.	Released	Browse decks · Repo guide
📬	Newsletter	Updates on new chapters, releases, and what the community is building.	Ongoing	Subscribe

Two earlier-stage experiments also live here. SocratiQ is an AI tutor that adds guided reading, quizzes, and spaced repetition to static learning sites. MLPerf EDU is a teaching benchmark suite aligned with MLCommons MLPerf.

---

Start Here

Where to begin depends on what you want from the curriculum.

| If you are... | Start here | Then go deeper | |---|---|---| | A student or self-learner | Read Volume I | Build TinyTorch, model systems with MLSys·im, try the labs (early release), and practice with StaffML | | An instructor | Open the Instructor Hub | Pick a course and use its syllabus, slides, kits, and labs | | A contributor | Find the guide for your component under Contributing | Report problems through GitHub Issues |

---

How the Material Is Made

No chapter is written once. Each one goes through a cycle that repeats as the field changes.

1. Watch the field. I follow the research and industry venues where new work appears: ISCA, MICRO, HPCA, ASPLOS, ISPASS, MLSys, NeurIPS, ICML, ICLR, Hot Chips, and industry summits. 2. Distill. A chapter keeps the principles and leaves out the variations. Quantization, for example, has been implemented in dozens of ways, and a student needs the principles that explain all of them. To decide what belongs, I use a test that David Patterson, who wrote the foreword to Volume I, gave me. A technique earns a place only when several companies use it in practice. That test ties the material to systems in production, so what a student learns carries over to practice. 3. Review with practitioners. I review chapters one-on-one with the engineers who build these systems inside large companies, from hyperscale infrastructure engineers to model developers. Practitioners at Arm, Google, Intel, Meta, Microsoft, Qualcomm, STMicroelectronics, and other organizations have reviewed the material, and academic reviewers in Europe, Asia, and the Americas have tested it against their own research and teaching. 4. Teach it. Volume I has been taught at Harvard through CS249r since 2023 and adopted by instructors at UC Berkeley, Columbia, Georgia Tech, IIT Bombay, and universities across Europe, Latin America, Africa, and Asia. The classroom shows where students get stuck. 5. Revise. Reader issues, reviewer notes, and classroom experience go back into the next draft.

Reviewers do more than check facts, and sometimes they change what the book covers. Sustainable AI and responsible AI both entered the book this way. Engineers building AI datacenters told me that the footprint of these facilities keeps growing and that students need to learn to account for it. Their concern led to a chapter I had not planned, which I wrote in 2023, before sustainable AI was a common topic.

Engineers in industry also urged me to teach responsible AI. It can look like a subject outside traditional engineering, but the systems in this book make decisions that affect people, and more and more of them act on their own. The engineers who build such systems are responsible for how they behave, so the book teaches that responsibility as part of the engineering.

Readers are part of the review, too. Every chapter is open, and when something is wrong, unclear, or missing, the book feedback forms send it straight into the next revision.

Numbers are computed, not typed. Every derived quantity in the books comes from Python code that ships with the text. I learned the need for this the hard way. In early drafts I typed numbers by hand, made mistakes, and watched the same quantity drift apart from one chapter and one volume to the next. No author can hold every number in four volumes in mind at once, so I wrote MLSys·im to hold them instead. It is the single source of truth for every chapter and every volume. Each hardware and model specification in it traces to a cited source, and when a value changes there, every chapter that uses it changes with it. A test suite runs on every commit and checks unit consistency, cross-references, notation, and citations.

---

FAQ

Why do you do all this?

When I began teaching machine learning systems, there was no textbook for the course. Books covered the models, and papers covered individual systems, but nothing taught the infrastructure underneath them as one subject. The field was changing too quickly for anyone to have written one. So I started writing it myself, because I have always learned best by building from scratch and teaching others.

When I want to understand an emerging or complex systems area, writing about it and writing code for it force me to slow down. They force me to find the principles that last, work through the math, and separate what is fundamentally true about physics and hardware from what is merely current.

This repository is my open workshop and teaching lab. It is where I work out ideas for my students at Harvard, my research lab, and open benchmarking efforts like MLPerf. Volumes III and IV are that process happening in public.

There is also a deeper reason. In the late 1990s, as an undergraduate, I wrote to a professor at Stanford because I was fascinated by genetic algorithms. I did not expect an answer. Then a brown package arrived at my house with a VHS tape on genetic algorithms inside, sent from the professor's office. I no longer remember the professor's name, so I never got to say thank you, but I have not forgotten what it felt like to have someone I had never met take the time to share what they knew with a curious student. That package sparked my curiosity, and the feeling has stayed with me.

Students write to me now the way I once wrote to that professor, far more than I can answer one by one. This curriculum is my way of sending the package back out, to a much bigger classroom than any lecture hall can hold.

Who is this for, and what should I know first?

It is for anyone who wants to understand how intelligent systems run, not only how models are trained: students, software engineers moving into ML infrastructure, hardware architects, and educators designing courses.

The only assumptions are that the reader can program in Python and has met basic machine learning ideas, such as a n