Deploy Your Own LLM Inference Service with vLLM

Deploy Your Own LLM Inference Service with vLLM

Hosting your own large language model is the fastest way to seize full control over latency, cost, and privacy. Instead of paying a per-token premium to an external vendor and praying that their rate limits hold, you can point an open-weight model at your own GPU and expose it behind a familiar API. In this tutorial you will deploy an open model as a production-grade inference service using vLLM. vLLM implements PagedAttention and continuous batching, two techniques that multiply throughput while keeping memory use low, which is why it has become the de facto engine for self-hosted inference. By the end you will have a live endpoint you can call from any OpenAI-compatible client.

Choosing the model and the right hardware

Before installing anything, make two decisions. First, pick a model. As a starting point use Qwen2.5-7B-Instruct or Llama-3.1-8B. Both run comfortably on a single consumer GPU and ship with many well-tested quantization formats. Second, know your GPU budget. On a 24GB RTX 4090, a 7B model with AWQ 4-bit quantization is the sweet spot: it fits in memory, leaves room for a generous KV cache, and still produces high-quality answers. On a smaller machine, start with a 3B or 4B model and trade a little quality for speed.

The simplest single-container install uses vLLM's official Docker image, which bundles CUDA, the engine, and the OpenAI-compatible server in one layer. As long as the NVIDIA Container Toolkit is installed on the host, launching a service is one command. The tool downloads the model weights from the Hugging Face Hub on first start, so make sure credentials are set if the model is gated. After the image pulls and the weights load, the server binds to port 8000 and prints the exact endpoint for chat completions.

docker run --runtime nvidia --gpus 1 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 --ipc=host \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 8192

Once the service is live, any client that speaks the OpenAI protocol can consume it. Set the base URL to http://localhost:8000/v1 and use any placeholder API key. Streaming works out of the box because vLLM sends tokens as server-sent events, which means a chat application can render words as they arrive instead of waiting for the complete response. The model field in every request is ignored or mapped to the served model, so swapping this endpoint into an existing codebase rarely requires changing a single line of business logic.

Validating the endpoint

Before tuning, prove the service actually works with a smoke test. Send a short prompt with a tiny max_tokens limit and confirm you receive a well-formed completion plus the expected usage counters. Then run the same request with streaming enabled and verify that tokens arrive incrementally rather than in one burst. A quick curl against the chat completions route is enough to catch misconfiguration before you spend time chasing through a client that never connected at all.

• Wrong base URL. The route must end in /v1; a missing path segment silently fails with a 404.
• Gated model without a token. Set HF_TOKEN or the mount will error on first pull.
• Port already bound. Check with docker ps before assuming the container failed to start.
• OOM on startup. Shrink the KV cache or switch to a quantized checkpoint.

Keep this smoke test in a script and run it after every change. Because vLLM exposes an OpenAI-compatible surface, the identical test works against any replacement engine, so your validation stays valid even if you later swap backends.

Tuning throughput and memory

The first variable worth tuning is --gpu-memory-utilization. vLLM by default reserves a large fraction of VRAM for the KV cache, but you may want to dial that down when the process shares the GPU with other workloads. The second lever is --max-num-seqs, which controls how many requests are batched at once. Raising it increases throughput at the cost of latency, because a longer batch means every request waits slightly longer while the whole batch completes.

PagedAttention is what makes vLLM memory-efficient: instead of storing a whole KV block per request, it stores readable blocks that can be shared and moved, mimicking how an operating system pages virtual memory.

Beyond the two headline flags, a handful of knobs moves the needle in predictable ways, and knowing what each one changes saves you hours of blind experimentation. Budget your tuning effort against the variables that actually matter for your traffic pattern rather than tweaking everything at once.

If your model is too large for the GPU, enable quantization with --quantization awq and pass a quantized checkpoint. Run the offline benchmark script that ships with the engine to see tokens-per-second before and after changes, and keep a baseline so you can judge whether a tweak actually helps. Monitor VRAM with tools such as nvtop; a healthy service should stay stable for hours without OOM restarts.

Securing and exposing the endpoint

Do not put the raw Docker port directly on the public internet. Put vLLM behind a reverse proxy such as Caddy or Nginx, terminate TLS there, and add an API key in front of the model. An OpenAI-compatible key can be enforced easily because the front-end server rewrites the incoming token and forwards the original request. Add a per-IP rate limit to absorb abusive traffic, and use Docker's memory cap so a runaway prompt cannot swallow every free byte on the host.

For horizontal scaling, run several vLLM replicas behind a load balancer. Each replica serves the same model and exposes the same OpenAI-compatible route, so clients notice nothing. Keep a warm pool sized to expected concurrency, and scale down after peak hours. If peak load regularly exceeds a single GPU model, look into split tensor parallelism across two cards, which vLLM supports natively via --tensor-parallel-size 2.

Observability and a production checklist

vLLM surfaces Prometheus metrics on a dedicated port, including request latency histograms and the number of running and waiting sequences. Hook those into Prometheus with Grafana and you will notice memory pressure and saturation before users complain. Finally, verify safety: add a guardrail on outputs, enforce a reasonable max_tokens, and log every request for audit. With these pieces in place, your self-hosted service will be fast, private, and dependable.