Why are local LLMs 'slow even though they fit in memory'? Understanding quantization, KV cache, and generation speed separately

The model loaded. There are no out-of-memory errors. Yet, you wait a long time for text to appear—with local LLMs, there is a ...

Running a 35B Parameter AI Model on iPhone at 11 Tokens per Second

Run a 35B parameter AI model locally on your iPhone using a Mixture of Experts architecture. The Flash iOS port hits 11 tokens per second.

VRAM usage doesn't decrease even with 4-bit quantization? What I learned from actual measurements with vLLM

"If you quantize a model to 4-bit, VRAM usage will be roughly 1/4"—this is a feeling you naturally develop when working with local LLMs. However, when I actually ran FP16, AWQ, and GPTQ on vLLM and ...
HackerNoon

Why 2-Bit LLM Quantization Fails at the Hardware Boundary

The jump to 2-bit LLM quantization isn't just about smaller integers. Explore the hardware-software boundary and why extreme compression requires co-design ...
Forbes

How Mixed-Precision Quantization Could Break AI’s Power Addiction

It turns out the rapid growth of AI has a massive downside: namely, spiraling power consumption, strained infrastructure and runaway environmental damage. It’s clear the status quo won’t cut it ...
InfoWorld

What is model quantization? Smaller, faster LLMs

Reducing the precision of model weights can make deep neural networks run faster in less GPU memory, while preserving model accuracy. If ever there were a salient example of a counter-intuitive ...
Unite.AI

Pinecone Open-Sources VQ-Bench Vector Quantization Framework

Pinecone has released VQ-bench, an open-source framework for building and benchmarking vector quantization methods, in a ...