Build Your Own AI Meeting Notes App: Whisper + LLM Summaries in Python

Build Your Own AI Meeting Notes App with Whisper and an LLM

Meeting-note tools are everywhere in 2026, but they all run in someone else's cloud, and none of them know which meetings matter to you. In this tutorial you will build your own: a small Python app that takes an audio file of a meeting, transcribes it locally with Whisper, summarizes it with an LLM into decisions and action items, and writes a clean Markdown note. Total code is under a hundred lines, it runs on any laptop, and when you finish you will understand exactly how tools like Granola and Otter work under the hood.

Here is the pipeline we will build:

1. Capture or import audio — any recording from your phone, laptop, or meeting app
2. Transcribe with Whisper — using faster-whisper for local, timestamped speech-to-text
3. Summarize with an LLM — a strict-JSON prompt that extracts decisions, actions, and open questions
4. Write a Markdown note — saved with today's date, ready for your notes folder

You will need Python 3.10 or newer, about 2 GB of free disk for the Whisper model, and an LLM API key. The transcription runs locally and free; only the summarization step calls an API, and you can swap in a local model via Ollama if you want the whole pipeline offline.

Step 1: Set Up the Environment

Create a project folder and a virtual environment, then install the two dependencies: faster-whisper for transcription and the openai client for LLM calls. Faster-whisper is a reimplementation of OpenAI's Whisper that runs several times faster on CPU thanks to the CTranslate2 engine.

# install with a virtual environment
python3 -m venv venv
source venv/bin/activate
pip install faster-whisper openai
# ffmpeg is required on the system for audio decoding
sudo apt install -y ffmpeg   # Debian/Ubuntu; brew install ffmpeg on macOS

If you are on macOS or Windows, use your package manager to install ffmpeg; on Windows, winget install ffmpeg works. You can verify the install with ffmpeg -version.

Step 2: Transcribe With Timestamps

Create a file called transcribe.py. The "small" model is a good balance of speed and accuracy for meetings; use "base" for faster runs or "medium" when the audio is noisy or full of jargon. The int8 compute type keeps memory low on CPU-only machines.

from faster_whisper import WhisperModel

model = WhisperModel("small", device="cpu", compute_type="int8")

def transcribe(path: str) -> list[dict]:
    segments, _info = model.transcribe(path, language="en")
    return [{"start": s.start, "end": s.end, "text": s.text.strip()}
            for s in segments]

segs = transcribe("meeting.m4a")
for s in segs:
    print(f"[{s['start']:7.1f}] {s['text']}")

Run it against any recording: python transcribe.py after pointing the function at your file, and you will see each spoken segment with a start time. If you plan to process hour-long meetings, consider using the "base.en" or "small.en" English-only variants, which are smaller and noticeably faster.

Step 3: Summarize With a Strict Prompt

Now for the part that turns a transcript into a useful note. Create summarize.py with a system prompt that demands structure. The key trick is strict JSON output: by forcing the model to return a schema, you can pipe the result straight into a Markdown renderer without fragile text parsing.

from openai import OpenAI
import json, os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
# For a fully local setup, point base_url at Ollama:
# client = OpenAI(base_url="http://localhost:11434/v1",
#                 api_key="ollama")

PROMPT = """You are a precise meeting summarizer. From the transcript
below, extract: (1) decisions, (2) action items with owners,
(3) open questions, (4) a three-sentence summary.
Return STRICT JSON with keys: summary, decisions, actions, questions.
Never invent items that are not in the transcript."""

def summarize(text: str) -> dict:
    resp = client.chat.completions.create(
        model="gpt-4.1-mini",  # or any OpenAI-compatible model
        messages=[
            {"role": "system", "content": PROMPT},
            {"role": "user", "content": text},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(resp.choices[0].message.content)

Notice what the prompt does and does not do. It does not ask for a creative recap; it defines four concrete buckets — summary, decisions, actions, questions — and forbids inventing items. That constraint matters more than the model choice: meeting notes are only useful if they are faithful.

Step 4: Glue It Together Into a Markdown Note

The final script reads the transcript, feeds it to the summarizer, and writes a dated note. For very long meetings, transcripts can exceed the model's context window, so chunk the text by timestamp blocks (for example every 1,500 words) and summarize each chunk, then ask the model once more to merge the partial summaries.

import datetime

def build_note(segs, summary: dict) -> str:
    date = datetime.date.today().isoformat()
    lines = [f"# Meeting Notes - {date}", ""]
    lines.append("## Summary")
    lines.append(summary["summary"] + "\n")
    lines.append("## Decisions")
    for d in summary["decisions"]:
        lines.append(f"- {d}")
    lines.append("\n## Action Items")
    for a in summary["actions"]:
        lines.append(f"- [ ] {a}")
    lines.append("\n## Open Questions")
    for q in summary["questions"]:
        lines.append(f"- {q}")
    return "\n".join(lines)

# wire it together
segs = transcribe("meeting.m4a")
raw = " ".join(s["text"] for s in segs)
result = summarize(raw)
with open(f"notes-{datetime.date.today()}.md", "w") as f:
    f.write(build_note(segs, result))
print("note written")

Run the whole thing with python meeting_notes.py meeting.m4a and you will get a file like notes-2026-09-08.md containing the summary, a bulleted decision list, action items with owners, and open questions. Add a cron job or a scheduled task, point it at your recording folder, and you have an automatic meeting-notes pipeline.

Pitfalls and Pro Tips

• Numbers and names get mangled. Whisper is good but not perfect; keep the audio quality high and consider the initial_prompt parameter with the names of participants and product terms to bias transcription.
• No speaker labels. Whisper alone does not do diarization. For "who said what," you need a diarization model such as PyAnnote; for most personal notes, timestamps are enough.
• Hallucinated actions are worse than none. If the LLM adds an action nobody agreed to, it will quietly drive work. Keep the "never invent" constraint and spot-check the output.
• Privacy is a feature. Because transcription is local, the raw audio never leaves your machine; only the transcript text is sent to the LLM provider. Delete audio files after processing if you are handling sensitive meetings.
• Respect consent. Check local laws and your team's policy before recording any call — even for personal notes.

Pro tip: summarize in chunks when meetings run long, then merge. A single giant prompt will either blow the context window or quietly drop the middle of the meeting.

Where to Go From Here

You now have a working meeting-notes app in about ninety lines of Python. Natural extensions: add speaker diarization, auto-detect action-item owners against your calendar, push notes to Notion or Obsidian through their APIs, or swap the summarizer for a local model so the entire pipeline runs offline. The architecture you just built — capture, transcribe, structure, store — is the same one behind commercial tools that charge $14 a month. The difference is that yours is private, free, and entirely under your control.