Andy Morrell

Homelab AI: Squeezing a 64k Context Window into 8GB VRAM (at 50 TK/s)

Andy Morrell ·

My last homelab update covered the initial setup of Ollama and Open WebUI. The stack was working, but running standard models left a lot of capability on the table. Recently, I set a specific goal: get Qwen 3.5 9B running locally with a massive 64k context window. The constraint was unyielding. My homelab runs on an 8-year-old NVIDIA RTX 2080 with exactly 8GB of VRAM. If you try to load a standard 9B model into VS Code via Continue while extending num_ctx to 64,000, you will instantly crash into an Out-Of-Memory (OOM) error. Here is exactly how I optimized the infrastructure, runtime variables, and container parameters to make it fast, stable, and running at a blistering 50 tokens per second.

1. Stripping the OS Overhead

The first win happened at the operating system layer. Because my homelab runs on a headless Ubuntu Server build, there is zero baseline VRAM consumption from a graphical desktop or background window managers. Starting with a completely clean GPU footprint gives me access to virtually the entire physical 8GB frame buffer.

2. The Math Behind the VRAM Budget

The base model load takes up exactly 6.4GB of VRAM, which leaves a highly restrictive 1.6GB budget to handle the active execution framework and the expanding conversation history. VRAM usage is split into model weights and the KV Cache (Key-Value Cache). As your context window grows linearly, the memory required for the KV cache escalates drastically. If you do not touch the underlying engine parameters, expanding to 64k tokens will bleed past your hardware limits and trigger a hard crash.

3. The Docker Environment Fixes

Instead of degrading the model weights with aggressive quantization, I left the base 6.6GB model intact and optimized the Ollama execution layer inside my Docker Compose config. By injecting three key environment variables, I forced the container to handle memory allocation with extreme efficiency:

environment:
  - OLLAMA_FLASH_ATTENTION=1
  - OLLAMA_KV_CACHE_TYPE=q8_0
  - OLLAMA_NUM_PARALLEL=1
  - OLLAMA_KEEP_ALIVE=-1
  • OLLAMA_FLASH_ATTENTION=1: Drastically reduces the memory overhead of the attention matrix as context scales up.
  • OLLAMA_KV_CACHE_TYPE=q8_0: Crucially quantizes the conversation cache to 8-bit, slicing the token memory cost in half. This compressed the entire 64k context footprint down into a tight 500–600MB envelope.
  • OLLAMA_NUM_PARALLEL=1 & OLLAMA_KEEP_ALIVE=-1: Dedicates the entire GPU compute pool to a single stream and locks the model persistently into VRAM to avoid reload lag.

4. The Result

Integrated directly into VS Code via Continue with a timeout: 600000 rule, the stack is completely stable. I can feed full repository structures, multi-page configuration files, and deep historical logs directly into a completely local model running on consumer hardware from 2018. Nothing leaves the house. It generates responses at a rock-solid 50 tokens per second. It proves that you do not need expensive corporate cloud compute allocations to build highly intelligent, context-aware agent loops. You just need to respect and optimize around your hardware constraints.