OpenLake slashes first‑token latency for long‑context models OpenLake moved KV cache storage from GPU RAM to a persistent Rust engine and saw a 66× speedup on time‑to‑first‑token for a 128K context window. The same en…

The announcement arrived on Hacker News alongside a paper titled GPU Offload in Rust: Portable, Safe, and Fast. The paper’s abstract promises a storage layer that is both portable and low‑overhead, matching the claims…

How OpenLake achieves low‑latency offload OpenLake is built on Rust and the Linux iouring interface. Rust provides memory safety without a garbage collector, and iouring lets the engine issue asynchronous I/O with min…

The engine sits between the GPU and host storage. During inference, the model writes its KV cache once, then reads it back in milliseconds from host RAM or disk. The path from storage to GPU memory stays short and pre…