Exercises
Work each one out on paper or with a calculator first, then expand the answer. Use the Cheat Sheet if you get stuck.
Warm-up: sizing
1. How much memory does a 14B model need at 4-bit, before the scratchpad?
Answer
14 × 0.6 ≈ **8.4 GB**, so about 9 GB. (28 GB would be the full 16-bit size.)2. How much does a 70B model need at full 16-bit precision?
Answer
70 × 2 = **140 GB**. That's why almost nobody runs 70B models at full precision at home.3. A 3B model at 8-bit, with 8K context and normal headroom. What’s the total?
Answer
Model 3 × 1 = 3 GB. A 3B model's scratchpad is smaller than an 8B model's, so roughly 0.3–0.5 GB. Plus ~1.5 GB headroom. Total ≈ **5 GB**.Concepts
4. Two machines both have enough memory to hold a model. What mainly decides which one generates text faster?
Answer
**Memory bandwidth.** Every word requires reading the whole model from memory, so tokens/s ≈ bandwidth ÷ model size.5. Your graphics card has 8 GB of VRAM and the model file is 7.8 GB. Will it run well?
Answer
**Probably not.** There's no room left for the scratchpad and software overhead, so it'll fail to load or spill into RAM. Pick a smaller quantization or a smaller model.6. A Mixture-of-Experts model is “30B total, 3B active”. At 4-bit, how much memory does it need to load?
Answer
**About 18 GB**, because memory follows the *total* count (30 × 0.6). The 3B active count only affects speed.Speed
7. A 70B model at 4-bit (~42 GB) on a desktop CPU with 64 GB of DDR5 RAM (~80 GB/s). How fast?
Answer
80 ÷ 42 ≈ 2, × 0.6 ≈ **1 token/s**. It fits, so it loads, but it's far too slow to feel interactive.8. An 8B model at 4-bit on an RTX 3090 (~950 GB/s). How fast?
Answer
950 ÷ 5 = 190, × 0.6 ≈ **~115 tokens/s**.Real decisions
9. You have a Mac with 32 GB of unified memory. A friend recommends a 32B model at 4-bit. Good idea?
Answer
32 × 0.6 ≈ 19 GB + ~2.5 GB extras ≈ 21.5 GB. The GPU can use ~22–24 GB of 32, so **it just fits**, but with little room for long contexts or other apps. A 14B model, or a 30B-A3B MoE model if you have the room, would be more comfortable.10. You have a 16 GB graphics card and want the smartest model that still runs at 20+ tokens/s. The card has ~450 GB/s bandwidth. What size should you aim for?
Answer
**Fit limit:** 16 − 1.5 headroom − 1 scratchpad ≈ 13.5 GB → 13.5 ÷ 0.6 ≈ ~22B at 4-bit. **Speed limit:** for 20 tokens/s, model size ≤ 450 × 0.6 ÷ 20 ≈ 13.5 GB. Same answer. So a model of **around 20B at 4-bit** is the ceiling. In practice, a 14B model gives you comfortable margin at ~30 tokens/s.Challenge
11. Using the exact KV cache formula from Lesson 3, compute the scratchpad for a model with 80 layers, 8 KV heads, head dimension 128, 16-bit values, at 8K context.