int2

Here are 2 public repositories matching this topic...

An innovative library for efficient LLM inference via low-bit quantization

SSD-resident INT2/INT4 KV cache with asynchronous staging and fused CUDA dequantizing decode attention for long-context LLM inference.

cuda pytorch ssd attention quantization nvme asynchronous-io gpu-kernels gqa kv-cache int4 long-context llm-inference int2 inference-systems

Add a description, image, and links to the int2 topic page so that developers can more easily learn about it.

To associate your repository with the int2 topic, visit your repo's landing page and select "manage topics."