Serve a local model as an API endpoint

Running a model in a notebook is a demo. Serving it behind a stable HTTP endpoint is what lets the rest of your stack actually use it. The good news: you almost never write the server yourself. A dedicated inference server — vLLM, llama.cpp's llama-server, Ollama, or SGLang — loads the weights once, keeps them warm in memory, and exposes a request interface. Better still, most of them expose the same interface: an OpenAI-compatible API, so a client you already have often works by pointing it at a new base URL. This walks the path from a loaded model to a callable endpoint, then the failure modes that bite under real traffic.

Prerequisites#

• A working compute stack (driver, toolkit, framework) if you're serving on a GPU, verified with a tiny workload first. On Apple Silicon there's no separate VRAM — the model shares unified memory with everything else; the same rules apply, you just count RAM.
• The model weights downloaded locally, with their license read. Size sets your floor: a quantized ~30B-class model (a Gemma or Qwen, say) fits on a 24GB card like an RTX 4090 or a well-specced Mac; a 4B model runs on a laptop.
• An inference server chosen for your model family and hardware — Ollama or LM Studio for a single-user laptop, vLLM or SGLang for a multi-user box. Prefer a maintained one over a hand-rolled loop.
• A way to test HTTP from the terminal, such as curl.

Step 1: Load the model once, not per request#

The single most important rule of serving: load the weights at startup and hold them in memory. Loading them means reading gigabytes off disk, moving them into VRAM, and initializing GPU kernels — seconds to tens of seconds of work. Do it per request and every call pays that toll; the classic symptom is a first call that takes many seconds and a second call exactly as slow. A real server pays the cost once and amortizes it across every request after.

# Start the inference server, pointing it at your local weights.
# It loads the model once and stays resident.
inference-server --model /path/to/model --host 127.0.0.1 --port 8000

Step 2: Confirm it is up before integrating#

Hit the health route first. If there's no health route, hit the lightest endpoint with a trivial input. Do this before wiring any app to it — a model that's still loading, or that failed to fit in memory, looks exactly like a broken client from the caller's side.

curl http://localhost:8000/health

Step 3: Send a real request#

Most servers speak the OpenAI-compatible API, so the real endpoint is usually /v1/chat/completions, not a bespoke route. Post a small payload and read the response. Keep the first test minimal so a failure points at the transport, not your prompt.

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "local-model",
    "messages": [{"role": "user", "content": "Say hello in one short sentence."}],
    "max_tokens": 32
  }'

The model field is whatever name the server registered the weights under — check /v1/models if you're unsure. The payoff of this being OpenAI-compatible: any client that already talks to OpenAI talks to your endpoint by swapping the base URL to http://localhost:8000/v1. You don't rewrite the app.

Step 4: Set sane limits#

An endpoint with no limits is a denial-of-service waiting to happen. Cap the maximum output length and the number of requests processed at once. The right concurrency number depends on your memory, not your CPU core count — and the reason is the KV cache: every in-flight request holds the model's running memory of the tokens it has seen so far, and that cache grows with each request's context length. Ten short requests and ten long ones cost wildly different amounts of VRAM, so the ceiling isn't "how many requests," it's "how much total context is live at once."

inference-server --model /path/to/model \
  --max-tokens 512 \
  --max-concurrent 4

Real servers name these limits differently — vLLM calls them --max-num-seqs and --max-model-len. Modern ones combine two distinct techniques: continuous batching (requests join and leave the running batch each step) plus paged KV-cache memory (vLLM's PagedAttention is the best-known example) to pack live requests into VRAM efficiently, so you set a couple of ceilings rather than a fixed batch size. The ceiling is still your VRAM.

Step 5: Put it behind a stable address#

Clients should call one stable URL, not the raw process port. Front the server with a reverse proxy so you can restart, swap, or scale the model without every caller changing its config. It's also where the things an inference server doesn't do well belong — TLS, authentication, and rate limiting.

Step 6: Add a timeout, and prefer streaming#

Generation latency varies with input and output length, so always set a client timeout — one slow request should never hang the calling app.

curl --max-time 30 http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "local-model", "messages": [{"role": "user", "content": "Summarize: ..."}], "max_tokens": 256}'

Better than a long fixed timeout: stream the tokens as they're generated ("stream": true on the OpenAI-compatible route, delivered as server-sent events). You get the first token fast, can show progress, and can time out on idleness — no token for N seconds — instead of guessing a total wall-clock budget for a response whose length you don't know in advance.

Pitfalls#

• Reloading per request. If your first and second calls are equally slow, you're loading the model every time. Use a server that keeps it resident.
• Unbounded concurrency. Each in-flight request holds a KV cache that grows with its context, so too many at once — or a few very long ones — causes out-of-memory crashes, not a graceful slowdown. The request that OOMs you is often the long one, not the tenth one. Cap both concurrency and context length to what your VRAM allows.
• No output cap. Without a max-tokens limit, a single request can run for a very long time and starve everything else.
• Assuming OpenAI-compatible means identical. The common path works, but servers differ on which sampling parameters, tool-calling, and response fields they support. Test the exact features your client uses, not just a hello-world.
• Binding to all interfaces by accident. Listening on 0.0.0.0 exposes the endpoint to the network. On a shared machine, bind to 127.0.0.1 unless you've put real auth in front.
• No timeout on callers. A hung generation with no client timeout looks like an app freeze. Set one everywhere you call the endpoint.
• Skipping the health check. Wiring three services to an endpoint that never actually started turns a one-line fix into a debugging session. Confirm the server is up first.

An inference endpoint is infrastructure. Treat it like any other service: one stable address, explicit limits, health checks, and timeouts. Once those are in place, swapping the model behind the URL — a newer Qwen, a smaller Gemma, a quantized build that frees up VRAM — becomes a config change rather than a rewrite.