deepinfer-swa

deepinfer/doc/inference_deployment_qa.md · a679bb8 · L235–245 · 固定 commit 上游

这是固定版本源码的只读摘录；可在线阅读，也可通过上游链接查看原文件。

235- How is recomputation done?
236
237    - SWA bounded replay recomputes only the most recent 128 tokens and truncates the attention window to within the replayed segment: replaying from position s, the query at position i is computed only against keys in the range max(s, i−127) to i, which yields an approximate SWA KV cache. The cost of recomputation is thereby reduced to a constant, independent of context length and layer count. The mechanism is used in two places:
238
239    - Encoder SWA bounded replay: the prefix cache hit length now depends only on the block KV cache. When the SWA KV cache is missing, the last 128 tokens of the hit prefix are recomputed and run forward together with the non-cached suffix; the replayed segment only rebuilds the SWA KV cache, while the block KV cache is reused as-is, neither recomputed nor overwritten.
240
241    - Decoder SWA bounded replay: prefill computation happens only in the encoder layers. In the CED architecture the decoder layers' block KV cache is projected from the last encoder layer's hidden state, so only the decoder SWA KV cache has to be computed per layer; reconstructing it exactly would require running the last 2,560 tokens through 20 decoder layers. Instead, each prefill recomputes only the last 128 tokens of the prompt in the decoder layers, and the result is used for decode only and never enters the prefix cache — nearly halving total prefill computation.
242
243- What is the hit policy?
244
245    - The principle is: the hit length is determined by the block KV cache, and whether the last block needs recomputation is determined by the SWA KV cache — if the SWA KV cache misses, back off by one block and recompute. Concretely, under the YOCO truncation requirement, the maximum hit length for an input of length L is at most L − 134 (128-token window + 6 tokens of speculative backtracking). Under that cap, the block KV cache is queried first, giving a hit length L1 (a multiple of the 128-token block size), and then the SWA KV cache is queried. If an SWA KV cache exists in the range L1 − 128 to L1, the hit length is L1; otherwise it is L1 − 128, where 128 is the recomputation window.