Decoupled Attention Fusion: Accelerating RAG with Efficient KV Cache Reuse

Retrieval-Augmented Generation (RAG) effectively mitigates hallucinations in Large Language Models (LLMs) but suffers from prohibitive Time-To-First-Token (TTFT) latency in long-context scenarios. Reusing pre-computed document KV caches addresses this but introduces a distribution mismatch, where offline caches lack the inter-document attention patterns required for coherent reasoning. CacheBlend reduces recomputation via selective attention, but suffers severe accuracy degradation at longer contexts.

To address these challenges, researchers propose Decoupled Attention Fusion (DAF), a framework that maintains high accuracy while significantly reducing recomputation overhead. DAF decouples the attention process into three integrated stages: important-token self-attention to restore missing inter-document attention, question-document self-attention for standard inference, and a state fusion that concatenates their outputs to synthesize the final hidden states. By decoupling these operations into dense patterns, DAF is natively compatible with Flash-Attention kernels, maximizing hardware utilization without requiring complex attention masks.

Decoupled Attention Fusion Mechanism

DAF operates by decoupling the attention process into three integrated stages:

• Important-token self-attention: Restores missing inter-document attention patterns.
• Question-document self-attention: Standard inference mechanism.
• State fusion: Concatenates outputs to synthesize the final hidden states.

Performance Improvements

Experiments show that DAF delivers up to 2 times speedup over CacheBlend and 5.6 times over full recomputation with vLLM on long-context benchmarks, without sacrificing accuracy.

Key Takeaways

• DAF significantly reduces recomputation overhead while maintaining accuracy.
• DAF is compatible with Flash-Attention kernels, maximizing hardware utilization.
• DAF achieves up to 2 times speedup over CacheBlend and 5.6 times over full recomputation.