How to deploy Qwen locally: Ollama for speed, vLLM for throughput

There are two honest ways to run Qwen on your own hardware. Ollama if you want a working endpoint in five minutes for one person. vLLM if several people, or an agent loop, will hit the same endpoint and you care about throughput and prefix caching.

Do not run vLLM to serve one interactive session. You will spend an afternoon on configuration to get the same tokens per second you had with Ollama.

Path one: Ollama

# install, then pull a mid-size coding model
ollama pull qwen3-coder:30b
ollama serve

# verify
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-coder:30b",
    "messages": [{"role":"user","content":"Write a Python function that parses this log format."}]
  }'

Ollama exposes an OpenAI-compatible endpoint at /v1, which means most clients work by pointing base_url at http://localhost:11434/v1. Context length defaults are conservative — raise num_ctx in a Modelfile if you need long files, and expect memory use to climb with it.

Practical notes:

• Keep the model resident (OLLAMA_KEEP_ALIVE) so you are not paying load time on every request.
• Quantised tags (q4_K_M) fit smaller GPUs but degrade instruction following first.
• On Apple silicon, unified memory changes the math: a 30B at 4-bit is comfortable on a 32 GB machine.

Path two: vLLM

vLLM is the right choice when the same model serves a team or an agent scaffold. The two settings that matter most for coding workloads are prefix caching and an explicit context cap.

pip install vllm
vllm serve Qwen/Qwen3-Coder-30B-A3B-Instruct \
  --served-model-name qwen3-coder \
  --tensor-parallel-size 1 \
  --max-model-len 65536 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching

Then point your client at http://localhost:8000/v1.

Why those flags:

• --max-model-len bounds KV cache. Leaving it unbounded is how you get an out-of-memory crash at 3am.
• --enable-prefix-caching is the single biggest win for agent loops, which resend the same system prompt and file context on every step.
• --gpu-memory-utilization 0.90 leaves headroom for the CUDA context. Push it to 0.95 and watch it fail on the second concurrent request.

Serving checks before you trust it

1. Cold vs warm latency. Measure the first request after a restart separately. It is dominated by weight loading and tells you nothing about steady state.
2. Concurrency curve. Fire 1, 4 and 16 concurrent requests. Throughput should rise, then flatten. If latency collapses instead, you are memory-bound and need a smaller model or a shorter context.
3. Long-context behaviour. Send a 30k-token request. If it truncates silently instead of erroring, your client is the problem, not the server.
4. Determinism. Run the same coding prompt five times at temperature 0.2. If the outputs differ wildly, your sampling config is drifting.

When to stop self-hosting

Self-hosting wins when you have idle hardware, strict data-residency requirements, or very high steady-state volume. It loses when your usage is spiky, when you need the frontier model rather than the open one, or when nobody on the team wants to own GPU operations.

A reasonable hybrid: serve a mid-size model locally for autocomplete and cheap edits, and call a hosted model from the Model Square for the hard tasks. Route by task class in your client, not by vibes.