Monitor Your LLM App in Production with LangSmith
Shipping a large language model app is the easy part; understanding what it does at 3am is the hard part. Traditional logging captures HTTP status codes but blinds you to the quality of a prompt, the drift of a model, or the reason a user received a weird answer. LangSmith fills that gap by giving every request a trace. In this tutorial you will wire up tracing, export metrics, and build dashboards that surface latency, token spend, and correctness regression long before your users file complaints.
Setting up tracing
LangSmith is built around traces, which record the full lifecycle of a request: the input, every tool call, the chain of model calls, intermediate reasoning, and the final output. Add the Python SDK to your project and configure the environment variables, and the instrumentation attaches automatically to popular frameworks such as LangChain, OpenAI clients, and native function calls. For a fast experiment, pass the project name and run a sample request; the event shows up in the web app almost immediately.
pip install langsmith
export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=lsv2_your_key
export LANGCHAIN_PROJECT=my-chatbot
Every trace has a trace ID and is composed of spans. Each span holds a specific operation, such as a single LLM call or a tool execution, plus its latency and token counts. Because spans are nested, the trace shows you the full path a request took, including which branch the model followed and how long each step waited. When a request is slow, open the trace and look for the widest span: that is where the seconds are going.
Tracking generation and correctness metrics
Latency is only part of the picture. In a chat app you also want generation efficiency, such as tokens per second and total input and output tokens, and correctness, which judges whether the answer actually satisfies the user. LangSmith lets you attach evaluators asynchronously after the trace completes. A correctness evaluator can use an LLM judge to grade the response against the expected answer, then store the score on the trace for later analysis.
Trace first, optimize second. You cannot fix a bottleneck you have never measured, so make sure every significant request is recorded before you tune a single prompt.
Do not stop at a single score. Attach several evaluators so one metric cannot hide another: a relevance judge checks whether the answer addresses the question, a toxicity filter flags unsafe output, and a formatting validator confirms the model returned the shape your code expects. Store each as a separate numeric field.
Token cost is a metric every budget holder cares about. Set up a model cost evaluator that multiplies usage fields by the pricing of the active model and stores the total on each trace. Over a day you can then answer questions such as which feature consumes the most tokens or which model variant is cheapest for a given task. Store the result as a numeric metric so it aggregates cleanly over all runs.
Instrumenting the tools you already use
LangSmith does not force you into a specific framework. If you speak to OpenAI directly, wrap the client calls with the built-in @traceable decorator and you get the same trace for raw HTTP calls that you would from a managed wrapper. If you use LangChain, the integration is automatic once tracing is enabled; chains and agents report their steps with no extra markup. The key is to start instrumented now rather than retrofitting later, because a trace laid down after the fact captures none of the behaviour you actually need.
Decide what a single trace should represent before you wire anything up. A good rule of thumb is one trace per end-user request end to end, so you can ask questions at the human level, such as whether a given answer drew on retrieval or hallucinated outright. Keep the span names stable, since you will filter on them in the dashboard, and add metadata such as a user ID or a feature flag to make every trace searchable in context.
- Start time and total duration, so slow requests stand out at a glance.
- Model name and version, so a rollout to a new checkpoint is identifiable in every trace.
- Prompt and completion tokens, the raw material for cost accounting.
- Final answer and confidence signals, so correctness checks have something to grade.
Once you adopt these fields, a single dashboard query can slice the data by user, by feature, or by model, which turns monitoring from a reactive firefight into a daily routine you actually trust.
Comparing runs and catching regressions
One of LangSmith's strongest workflows is the experiment, a batch of annotated inputs run against two model versions or two prompt templates. Each experiment produces a comparison of metrics, revealing whether a new prompt improved correctness or silently raised latency. Run these experiments before every deployment so a model upgrade never ships a regression. If a score drops, the diff view shows the exact prompt and response pair that changed under the new configuration.
Pair numeric metrics with a readable trace whenever you drill into a problem. A dashboard shows you that p95 latency spiked at 14:00, but only the trace explains it, whether a retry loop doubled a tool call or a slow embedding call throttled the chain. Keep the two views linked by storing the trace ID in every alert payload, so the person who receives the alert can jump straight to the evidence instead of searching.
Finally, protect your historical trend. Freeze experiment datasets and keep their inputs stable, because a changing eval set quietly invalidates every comparison you make. When you retire an old model or prompt, keep the old traces archived rather than deleting them; a regression six months later is much easier to explain when the earlier runs are still inspectable side by side with the current ones.
Production monitoring habits
A monthly experiment is too rare to catch slow drift, so add scheduled monitoring that runs your evaluation set nightly and alerts on threshold crossings. Watch the error rate, p95 latency, and average correctness together: a correctness drop combined with rising latency often signals an overloaded or degraded model. Archive old traces to control storage costs, and keep your project name stable so historical comparisons remain valid. With a live tracing pipeline, your LLM app gains the observability it deserves, and debugging becomes a search through traces instead of a prayer.
