SystologyCaching
- Local LLM Inference
—
Caching
,
Machine-Learning
,
Optimization
and +1 more
An empirical systems exploration of local LLM inference on unified memory architectures: profiling compute-bound prefill vs. memory-bound decode phases, analyzing KV cache retention during cancellation, measuring the 64K vs. 128K context cliff on Apple Silicon, and comparing client payload strategies.