These are learning notes on quantization — the technique that lets multi-billion-parameter AI models run on consumer hardware instead of a data center.
1. Bits and bytes — the basics
A bit is the smallest unit of information: 0 or 1, like a switch.
A byte = 8 bits grouped together = 256 possible combinations.
1 bit = 2 choices = 2¹
4 bits = 16 choices = 2⁴
8 bits = 256 choices = 2⁸
16 bits = 65,536 choices = 2¹⁶
Why 2ⁿ? Each extra bit doubles the number of possible combinations — like a multiple-choice menu: 3 questions with 2 options each gives 2×2×2 = 8 combinations.
Memory sizes, for reference:
1 byte = 8 bits
1 KB = 1,000 bytes
1 MB = 1,000 KB
1 GB = 1,000 MB
2. The number formats used in AI
An AI model is really just billions of numbers, called "weights." The format you store them in determines both how precise each number is and how much space it takes up.
| Format | Bits | Bytes | Possible values | Typical use |
|---|---|---|---|---|
| FP32 | 32 | 4 | ~4.3 billion | Training, "high quality" reference |
| FP16 / BF16 | 16 | 2 | 65,536 | Fast inference, good compromise |
| INT8 | 8 | 1 | 256 (-128 to 127) | Moderate compression |
| INT4 | 4 | 0.5 | 16 (-8 to 7) | Ultra compression |
FP = floating point — a number with a decimal point (3.14, -2.71...). INT = integer — a whole number, obtained by rounding.
FP16 vs INT8/INT4 — not the same logic
- FP16 reduces precision (fewer decimal places) but keeps an enormous range of possible values.
- INT8/INT4 reduces the number of possible values — every number has to be forced into a small, fixed list (say, 256 or 16 values).
Concrete impact on model size
For an illustrative 13-billion-parameter model (≈26 GB in FP16):
FP32 : 13B × 4 bytes = 52 GB
FP16 : 13B × 2 bytes = 26 GB
INT8 : 13B × 1 byte = 13 GB
INT4 : 13B × 0.5 byte = 6.5 GB
Each step down roughly halves the footprint — which is exactly why quantization is what makes it possible to run large models outside of a data center.
3. Why INT8 = 256 values, INT4 = 16 values
With n bits, each either 0 or 1, the total number of combinations is 2ⁿ.
INT4 = 2⁴ = 16 values (0000 to 1111)
INT8 = 2⁸ = 256 values (00000000 to 11111111)
Every extra bit doubles the number of possible combinations — that's why the gap between 4 and 8 bits is enormous (16 → 256, not just "double").
4. The quantization process, step by step
Quantizing = rounding numbers that already exist so they fit in fewer bits.
Step 1 — Measure the range of the original weights
Look at the minimum and maximum values in the model (e.g., -5.2 to +8.7).
Step 2 — Compute the scale factor
scale factor = (max - min) / number of available values
Example for INT8: (8.7 - (-5.2)) / 256 ≈ 0.055
Step 3 — Round each number
quantized number = ROUND(original number / scale factor)
Example: 3.14159265 / 0.055 ≈ 57.1 → 57
Step 4 — Save the result
The new, smaller model is now ready to be loaded.
The two main approaches
| Method | Description | Advantage | Drawback |
|---|---|---|---|
| PTQ (Post-Training Quantization) | Quantize an already-trained model, no retraining | Fast (minutes) | Can lose a bit of quality |
| QAT (Quantization-Aware Training) | The model is retrained while simulating quantization | Better quality | Slow (days), expensive |
Most consumer-facing tools use PTQ — it's the pragmatic default when you're starting from someone else's trained weights.
5. Where do the weights to quantize actually come from?
The key point: nobody computes the weights from scratch. They already exist, trained by whichever organization built the model (Google, Meta, and so on), and stored in .safetensors files.
THE LAB (months of training)
Random weights → training on billions of tokens → final weights
→ saved in BF16 as .safetensors files
→ published on Hugging Face
QUANTIZERS (a few hours of work)
1. Download the original files
2. Open each file, read the already-trained numbers
3. Round each number to the target format (Q4, Q8...)
4. Save as .gguf (or another quantized format) files
→ published on Hugging Face
YOU
Download the already-quantized file directly
→ no computation on your end, just a download
In simplified code, the actual rounding step looks like this:
model = load_model("some-lab/some-model") # already-trained weights
for layer_name, weight_tensor in model.named_parameters():
min_val, max_val = weight_tensor.min(), weight_tensor.max()
scale = (max_val - min_val) / 16 # 16 = possible values in INT4
quantized_weight = round(weight_tensor / scale)
save(layer_name, quantized_weight, scale)
6. Common quantization tools and methods
| Tool | Principle | Strengths |
|---|---|---|
| Bitsandbytes | Quantizes on the fly at load time | Very simple, no preparation needed |
| GPTQ | PTQ with second-order mathematical optimization | Good compromise, widely supported |
| AWQ | Protects "important" weights, compresses the rest harder | Better quality, 2-3× faster inference |
| GGUF | A file format (not an algorithm) used by llama.cpp/Ollama; can hold different quantization levels (Q4, Q5, Q8...) | Easy to use locally (Ollama, LM Studio) |
| NVFP4 | A 4-bit format optimized specifically for recent NVIDIA GPUs | Very efficient on modern NVIDIA hardware |
Decoding the GGUF naming scheme (e.g., Q4_K_M)
Q4_K_M
│ │ │
│ │ └─ M = Medium (quality variant: S/M/L)
│ └─── K = "K-quant", mixes precision levels based on weight importance
└───── 4 = ~4 bits per number on average
K-quant is smarter than uniform rounding: it keeps more bits for the weights that matter most and compresses the rest harder — similar in spirit to AWQ.
7. Analogies that stuck
- Quantization is like compressing a photo: you lose a bit of detail, but the file is much smaller and loads much faster.
- FP32 → INT4 is like summarizing a book: the original manuscript (FP32) becomes a condensed summary (INT4) — still readable, just less detailed.
- Unsloth/AWQ/GPTQ act like an editor: they take the "manuscript" (the original weights from a lab) and publish a condensed, ready-to-use version. You just download the finished result.