4 — Memory & batching

Batching turns memory-bound into compute-bound

Emmanuel Pilliat — ENSAI 3A

Goals

• The CPU spends most of its time waiting for memory — we measure the cache hierarchy.
• Locality: the same maths, walked in the wrong order, runs 11.9× slower — and we prove the cache is the culprit.
• Batching: stacking examples turns a memory-bound problem into a compute-bound one — measured on the CPU, before any GPU appears.

The compilation module made one core fast; the threads module used many. Today’s enemy is neither — it is the distance to your data.

▶ Notebook (demo + solution): Memory & GPU — one notebook for this deck and the GPU deck.

⬅ Previous: 3 — Threads & concurrency · Next: 5 — GPU ➡

Part A — the CPU waits for memory

The memory hierarchy

• A core adds two floats in < 1 ns. Fetching one value from RAM takes ~100 ns.
• In that time the core could have done hundreds of operations. If it waits, it does nothing.

Level	Typical size	Latency	Analogy (if L1 = 1 s)
registers	~a few hundred B	~0	instant
L1	32–64 KB / core	~1 ns	1 second
L2	256 KB–1 MB	~4 ns	4 seconds
L3	8–32 MB	~15 ns	15 seconds
RAM	16–64 GB	~100 ns	~2 minutes

• A cache survives on a bet — locality: temporal (just used → will reuse soon) and spatial (used it → will use its neighbours → memory moves in lines of 64 B = 8 Float64).
• The whole art of fast code is honouring that bet.

This machine: L1d 544 KiB total (14 instances), L2 18 MiB (9 instances), L3 24 MiB shared. Hybrid CPU (P/E/LP-E cores) — per-core L1 differs, hence “instances”.

Bet before you look 🎲

We sum the same array over and over, growing its footprint from 8 KB to 64 MB.

The table says: L1 ≈ 1 ns per access, RAM ≈ 100 ns.

How much slower is the 64 MB version, per access?

(~100×? Write your number down.)

The cache cliff, measured

function repeated_sum(a, passes)  # footprint = a, accesses ×passes
    s = zero(eltype(a))
    for _ in 1:passes
        @inbounds @simd for i in eachindex(a)
            s += a[i]
        end
    end
    return s
end

footprint	ns / access	where it lives (this machine)
8 KB	0.049	L1
4 MB	0.128	L2 (18 MiB)
64 MB	0.317	RAM — spills the 24 MiB L3

The steps are real: you can see the hierarchy appear. But the cliff is 6.5×, not ~100× — why?

Absolute values are machine- and load-sensitive: run at rest and look for the JUMPS, not the exact ns. Full sweep (8 KB → 64 MB) is in the notebook.

Why 6.5×, not 100×? Bandwidth vs latency

• The table quotes latency: the wait for one isolated access.
• Our loop is a @simd sequential stream — the address pattern is perfectly predictable.
• So the prefetcher fetches lines before we ask for them: the latency is overlapped, and we pay something closer to the bandwidth cost.

The 100× latency is real — a predictable access pattern hides it. Random pointer-chasing has no pattern to prefetch: it pays the full ~100 ns per hop.

Remember this: the memory system rewards predictability. “Locality” is the code word for it.

Anti-oversimplification: this is NOT “the table is wrong” nor “machine-specific noise”. Latency and bandwidth are different resources; a streaming sum is bandwidth-bound, a linked-list walk is latency-bound. Same hardware, 15× apart.

Two walks, same maths

Sum every element of an 8000×8000 matrix — two loop orders:

# COLUMN by column
for j in axes(M, 2)
    for i in axes(M, 1)
        s += M[i, j]
    end
end

# ROW by row
for i in axes(M, 1)
    for j in axes(M, 2)
        s += M[i, j]
    end
end

• Julia stores matrices by columns (column-major): A[i,j] and A[i+1,j] are neighbours in memory.
• That is the opposite of numpy / C (row-major). Carry your numpy reflexes over unchanged and you go against the grain every time.

11.9× for reordering two loops

8000² Float64 ≈ 512 MB — far beyond any cache:

walk	time	each 64 B cache line is…
columns (with the grain)	30.4 ms	fully used: all 8 Float64
rows (against it)	361.4 ms	used for one element, then evicted

• Same result, same number of additions — 11.9× slower for the walk order alone.
• Julia rule: the loop over the first index (rows) is the inner loop — the opposite of C/numpy.

Getting it wrong doesn’t give a wrong result — just silently slow code, which is sneakier.

The notebook’s prose says “2 to 10×” — the measurement says 11.9×. Quote the measurement. (Ratio is size- and machine-dependent, but don’t understate it.)

Prove it’s the cache 🔬

If the gap were “Julia prefers columns”, it would show at any size. It doesn’t:

size	footprint	rows / columns ratio
100²	78 KB — fits in cache	0.9× — no gap at all
4000²	122 MB — spills	7.2×

• Small matrix → everything is already “close” → the walk order cannot matter. And measurably, it doesn’t.
• The gap appears exactly when the data outgrows the cache.

The claim is not “columns are fast”. It is: the cache decides — and this experiment isolates it.

Strongest argument of the module: it rules out every explanation except the cache. In the notebook this is the interactive slider; the script prints the full table (100² → 4000²).

numpy’s twin: a library can hide it, not remove it

Same experiment in numpy, C-order vs F-order layout (8000²):

numpy	ratio	why
A.sum(axis=0), C vs F	≈ 1×	nditer reorders the walk into memory order
np.copyto(C, F) vs C, C	13.2× (109.5 vs 8.3 ms)	a layout-mismatched copy cannot be reordered

• numpy protected you on sum: someone wrote the reordering for you.
• Step outside what the library can rearrange — a mismatched copy, a hand-written loop — and the full hardware penalty is back.

The penalty is identical in both languages. numpy hides it; Julia lets you see it. Neither removes it.

Measured: sum(axis=0) C/F = 1.14× at 8000², 0.77× at 2000² — noise around 1×, and load-sensitive. Don’t over-read either side of 1×. Demo lives in the Python notebook (python/4), which every student runs.

Part B — batching: memory-bound → compute-bound

Arithmetic intensity: FLOP per byte

How much work do you do per byte you load?

operation (n×n, Float64)	FLOP	bytes loaded	intensity
mat × vec	2n²	8n² (all of W)	0.25 FLOP/byte — constant
mat × mat	2n³	8n²	n/4 FLOP/byte — grows with n

• Low intensity → the compute units wait for loads: memory-bound.
• High intensity → each loaded value is reused many times: compute-bound.

⚠ Don’t compare their seconds — they do different amounts of work. The honest unit is GFLOP/s: work per second. One number, and it says by itself which resource is saturated.

Intensity is per BYTE, not per value (8× apart for Float64) — be consistent. 0.25 FLOP/byte is the roofline number for Float64 mat-vec. BLAS-3 (GEMM) tiling is what realises the n/4: each tile of W is reused from cache.

The thesis: batching = mat×vec → mat×mat

• A neural net is essentially matrix products. Inference on one example is W * x: every weight is loaded, used once, evicted. Memory-bound.
• Stack B examples as columns of X and W * X reuses each loaded weight B times. That is a mat×mat. Compute-bound.

y = W * x     # 1 example : each weight used once
Y = W * X     # B examples: each weight reused B times

Batching is not a GPU trick. It changes the arithmetic intensity of the problem itself. If that’s true, it must already pay on the CPU — let’s measure it.

Batching, measured on the CPU

Dense layer W (4096 → 4096, Float32), B examples as columns:

B	total	µs / example	GFLOP/s
1	2.55 ms	2551.8	13.1
8	5.65 ms	706.0	47.5
64	11.06 ms	172.9	194.1
256	39.06 ms	152.6	219.9
1024	141.80 ms	138.5	242.3

• Per example: 2551.8 → 138.5 µs = 18.4× better — for the same maths per example.
• Throughput: 13.1 → 242.3 GFLOP/s = 18.5× more of the CPU actually used.

Batching pays on the CPU, today, no GPU in sight. The GPU deck pushes the same idea much further.

Honesty check (measured): B=1 here is a skinny GEMM (W * X[:,1:1]) at 2.55 ms; a true gemv (W * x) is 1.09 ms — the B=1 baseline is 2.3× pessimistic. Against the honest gemv baseline the per-example gain is still ~7.9×. Say it if asked; the thesis stands either way. Measured cold, default BLAS threads. Never claim “speedup ≈ number of threads” — the threads module disproved it.

Why B = 1024 barely helps any more

Per example: 2551.8 → 172.9 (B=64) → 152.6 (B=256) → 138.5 µs. The curve flattens. Why?

• W is 4096² × 4 B = 64 MiB — it can never sit in the 24 MiB L3. Every batch must stream W from RAM: that is the fixed cost.
• Batching divides that fixed cost by B: time/example ≈ compute/example + (streaming W) / B.
• By B ≈ 256 the W-traffic per example is already tiny — further doubling buys almost nothing.

Flattening is the success signal: the memory-bound → compute-bound conversion is complete. Past that point, bigger batches only add latency and memory — the classic serving trade-off.

242 GFLOP/s is what this run reached, not the machine’s absolute peak — don’t present it as a hardware ceiling. The 1/B amortisation model fits the table: 138.5 µs ≈ pure compute per example; 2551.8 ≈ that plus the whole W stream.

Going further

• ▶ Notebook (demo + solution): Memory & GPU — sections 1–3 are this deck; the GPU sections (coalescing, CPU vs GPU, π on the card) belong to the GPU deck.
• 🐍 The numpy side (copyto, sum(axis=0), batching): python/4_memory_and_gpu_python.ipynb
• 🎯 The one to remember: batching is not a GPU trick — it converts memory-bound into compute-bound, and it already pays ~18× on a laptop CPU.

⬅ Previous: 3 — Threads & concurrency · Next: 5 — GPU ➡