Self-Hosting Open-Source LLMs: High-Throughput Inference with vLLM, PagedAttention & Ollama
A practical guide to deploying open-source AI models (Llama 3.3, Mistral, Gemma 2) on Linux and local hardware: comparing vLLM's PagedAttention throughput with Ollama's local developer workflow.
- 1.Why Self-Host Open-Source Models?
- 2.1. Local Developer Workflow: Zero-Config Deployment with Ollama
- 3.2. Production High-Throughput Inference with vLLM & PagedAttention
- 4.3. Tool Comparison: Ollama vs. vLLM vs. TGI
- 5.4. Quantization Demystified: GGUF vs. AWQ vs. GPTQ
- 6.5. Production Nginx Reverse Proxy Architecture
- 7.6. Key Takeaways for Production Deployments
Why Self-Host Open-Source Models?#
- Strict Data Sovereignty: Zero user data transmitted to third-party cloud providers.
- Fixed Infrastructure Costs: Predictable GPU server pricing rather than pay-per-token pricing at massive request volumes.
- Zero Rate Limits: Unlimited throughput constrained only by dedicated GPU hardware.
- Custom Quantization & Offline Execution: Running models in isolated air-gapped VPCs or edge hardware.
This tutorial contrasts two premier self-hosting tools: Ollama for local development and vLLM for high-concurrency production inference.
1. Local Developer Workflow: Zero-Config Deployment with Ollama#
Ollama bundles model weights, runtime execution, and GPU quantization into a clean, Docker-like CLI.
Quick Start on macOS / Linux:
# 1. Install Ollama
curl -fsSL https://ollama.com/install.sh | sh
# 2. Run Google Gemma 2 (9B parameter model with 4-bit quantization)
ollama run gemma2:9b
# 3. Pull Llama 3.3 70B for high-capability local reasoning
ollama pull llama3.3:70bIntegrating with Node.js via OpenAI-Compatible API: Ollama exposes a drop-in OpenAI-compatible REST server on port `11434`:
import OpenAI from "openai";
// Connect standard OpenAI SDK directly to local Ollama daemon
const localAI = new OpenAI({
baseURL: "http://localhost:11434/v1",
apiKey: "ollama", // Required by SDK but ignored by Ollama
});
async function askLocalModel() {
const completion = await localAI.chat.completions.create({
model: "gemma2:9b",
messages: [
{ role: "system", content: "You are a senior Linux systems engineer." },
{ role: "user", content: "Explain how PagedAttention reduces VRAM fragmentation in LLM inference." }
],
temperature: 0.2,
});
console.log(completion.choices[0].message.content);
}2. Production High-Throughput Inference with vLLM & PagedAttention#
While Ollama is optimized for single-user interactive use, vLLM is engineered for high-concurrency production workloads handling hundreds of parallel requests.
The Problem: Key-Value (KV) Cache Memory Fragmentation During LLM generation, intermediate key-value attention tensors are stored in GPU VRAM for every active request. Traditional inference frameworks allocate contiguous memory blocks, leading to **60–80% wasted GPU memory** due to memory fragmentation and unpredictable generation lengths.
The Solution: PagedAttention vLLM implements **PagedAttention**, inspired by operating system virtual memory paging: - Divides the KV cache into small, non-contiguous physical memory blocks. - Allocates memory dynamically as tokens are generated. - Increases serving throughput by **2x to 4x** compared to standard HuggingFace pipelines.
Launching vLLM in Production with Docker:
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model google/gemma-2-9b-it \
--tensor-parallel-size 1 \
--max-model-len 8192 \
--gpu-memory-utilization 0.903. Tool Comparison: Ollama vs. vLLM vs. TGI#
| Metric | Ollama | vLLM | HuggingFace TGI |
|---|---|---|---|
| Primary Focus | Local dev, desktop, prototyping | High-throughput production serving | Production enterprise serving |
| Memory Engine | llama.cpp (GGUF quantization) | PagedAttention (Dynamic VRAM paging) | FlashAttention-2 + Paged Attention |
| Multi-GPU Tensor Parallelism | Limited | Native (--tensor-parallel-size N) | Native |
| Concurrent Throughput | Low (Sequential priority) | Extremely High (Continuous Batching) | Very High |
| Ease of Setup | 1-line CLI command | Docker / Python virtualenv | Docker container |
4. Quantization Demystified: GGUF vs. AWQ vs. GPTQ#
Running 70B parameter models in full 16-bit precision requires over 140 GB of VRAM. Quantization reduces weight precision from 16-bit floating point (fp16) to 4-bit or 8-bit integers (int4 / int8) with negligible loss in accuracy:
| Format | Best Runtime | GPU Requirement | Precision / Quality | Ideal Use Case |
|---|---|---|---|---|
| GGUF (Q4_K_M) | Ollama / llama.cpp | CPU + Apple Silicon / Single GPU | High (Minimal perplexity loss) | Desktop & local developer workflows |
| AWQ (Activation-aware) | vLLM / TGI | NVIDIA GPU (Turing / Ampere / Hopper) | Highest (Optimized for activations) | High-concurrency production serving |
| GPTQ (4-bit) | vLLM / AutoGPTQ | NVIDIA GPU | High | Batch offline inference pipelines |
| FP8 (8-bit Float) | vLLM (Ada / Hopper H100) | NVIDIA RTX 4090 / H100 / L40S | Near-Zero Degradation | Enterprise data centers |
5. Production Nginx Reverse Proxy Architecture#
When hosting vLLM behind a microservices architecture, place a hardened Nginx reverse proxy in front to handle SSL termination, rate limiting, and health monitoring:
upstream vllm_backend {
server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
keepalive 32;
}
server {
listen 443 ssl http2;
server_name ai-inference.internal.company;
ssl_certificate /etc/ssl/certs/ai-cert.pem;
ssl_certificate_key /etc/ssl/private/ai-key.pem;
# Enforce streaming compatibility for Server-Sent Events (SSE)
location /v1/chat/completions {
proxy_pass http://vllm_backend;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
proxy_buffering off; # Essential for real-time token streaming
proxy_cache off;
proxy_read_timeout 300s;
}
location /health {
proxy_pass http://vllm_backend/health;
}
}6. Key Takeaways for Production Deployments#
- Use Ollama for Dev, vLLM for Production: Ollama delivers the fastest time-to-first-token for individual developers, while vLLM provides the continuous batching throughput needed for multi-tenant applications.
- Quantize to 4-bit AWQ or 8-bit FP8: You can run a 70B parameter model on a single 48GB GPU (like an NVIDIA RTX 6000 Ada or dual RTX 3090s) with 4-bit AWQ precision.
- Disable Proxy Buffering for SSE: When building streaming chat UIs, always set
proxy_buffering offin Nginx to ensure tokens stream to users with zero latency!
Published by
Vyuhantrix Team
AI & Systems Engineering · Vyuhantrix
Vyuhantrix is an open technology learning platform based in Ahmedabad, India, publishing step-by-step programming tutorials, system design breakdowns, and free developer tools.
Keep Learning
Recommended Guides
Google Gemini 1.5 Pro & Flash API: Multimodal Prompting, Function Calling & Structured JSON with Node.js
Master Google Gemini 1.5 Pro and Flash in Node.js. Learn multimodal image & video inputs, native structured JSON outputs, system instructions, function calling, and token budgeting.
Google AI Studio vs. Vertex AI: Choosing the Right Google AI Platform for Developers
Compare Google AI Studio and Google Cloud Vertex AI. Learn pricing models, IAM security boundaries, private VPC deployment, enterprise SLA guarantees, and model tuning.
Google Gemma 2: Running and Fine-Tuning Open AI Models Locally with Ollama and Python
A hands-on guide to Google's open-weights Gemma 2 models (2B, 9B, 27B). Learn local inference with Ollama, 4-bit quantization, and LoRA fine-tuning with Hugging Face.