Store KV cache of computed prompts to disk to avoid re-compute in follow-up runs #64
Description
Idea from: #23 (comment)
We can add a --cache_prompt flag that if added will dump the computed KV caches of the prompt processing to the disk in a file with name produced by the hash of the prompt. Next time you run, it will first check if we have stored KV cache for this hash and load it straight from disk instead of computing it.
Great task for contributing to the project!
Piezoid, blackhole89, rkrp, nleroy917, thomasantony and 24 more