An interactive edition of the Scaling Book
All About Rooflines
Part 1 of How To Scale Your Model · Introduction · Next: TPUsNext: GPUs
When we run algorithms on hardware, we're bounded by three things: how fast our computer can do math (OPs/second), the bandwidth available for moving data around (bytes/second), and the total memory available to store data (bytes). These "roofline" constraints let us upper and lower bound the time of a given computation.
Whose words are you reading? The prose comes from the original essay, with local changes for the selected hardware and HBM or network bandwidth. ✦ adaptation marks new paragraphs and sections written with OpenAI Codex. Live numbers and interactive figures replace fixed examples. Sources and editorial changes.
Where Does the Time Go?
Let's start with an extremely simple question: why does an algorithm take 50ms instead of 50s or 5ms? What is actually happening within the model that takes substantial time and how long should we expect it to take?
Computation: A deep learning model is effectively a bunch of matrix multiplications, each composed of floating-point multiplication and addition 'operations' (FLOPs). Our accelerator speed determines how long these take to compute. Let C be the compute rate per chip:
For instance, can perform about in bfloat16, a 16-bit floating point format often used in ML. That means doing 1012 operations will take roughly 1012 / C = at the current compute rate.✦ adaptation — Peak versus achieved throughput. This is a lower bound on compute time. Small operations, launch overhead, and kernels that use the hardware inefficiently can take longer. The compute multiplier lets you model a lower usable rate.
✦ adaptation — Published rates are theoretical peaks; real kernels generally achieve less. The compute and bandwidth multipliers model achieved rates as a fraction of peak (0.8 means 80%), or hypothetical faster hardware above 1. Compute also depends on the selected precision. All compute rates here are dense, without structured sparsity.
✦ adaptation — The native dense FP8 Tensor Core rate is per GPU. FP8 specifications.
Communication within a chip: Within an accelerator, tensors need to be transferred between accelerator memory (HBM) and the compute cores. You'll see the bandwidth of this link referred to as "HBM bandwidth." On , this is . Let W be this bandwidth in bytes/s and Q the bytes read and written.
Communication between chips: When we distribute a model across multiple accelerators, tensors frequently need to be transferred between them. There are often a few options for this on our hardware (ICI, DCN, and PCIeNVLink, the scale-out network, and PCIe), each with different bandwidths. Let W be the outgoing bandwidth per chip and Q the bytes sent by each chip.
We measure this in bytes/s and estimate the total communication time with:
Typically (but not always), computation within a single chip can be overlapped with communication within a chipbetween chips. This means we can lower-bound training and inference time by using the maximum of computation and communication time. We can also upper-bound with their sum. In practice, we optimize against the maximum as the algebra is simpler and we can usually come close to this bound by overlapping our communication and computation.
✦ adaptation — A matmul as a linear map
A matrix multiplication maps a token's vector from one latent space to another. A batch applies the same linear map to many token vectors.
- B
- Batch size: token vectors, not sequences.
- D
- Input width: — the input hidden size. D is split evenly between two chips.
- F
- Output width: — the output hidden size.
Call the input matrix X, the weight matrix Y, and the output matrix Z. Each row of X is an input vector; each row of Z is its transformed output. The multiplication contracts over D:
Hidden sizes in a Transformer
In a Transformer's feed-forward network, the up-projection maps the model hidden size D to the intermediate size F. The down-projection reverses these roles. A square weight matrix has D = F.
BF16 uses two bytes per element; INT8 uses one. Integer arithmetic is counted in OPs, floating-point arithmetic in FLOPs.FP8 uses one. Floating-point arithmetic is counted in FLOPs. One multiply-add counts as two operations.✦ adaptation — Quantization assumptions. These byte counts omit conversion work, quantization scales, and padding. Lower precision can also affect accuracy. We assume a 2× INT8 rate for the TPU exercise.
Let w be bytes per weight, a bytes per input activation, and o bytes per output. For , w = , a = , o = . The matmul performs approximately 2BDF operations and transfers Q = aBD + wDF + oBF bytes, assuming each input is read once and each output written once.
The matrices are split along D. Each chip does BDF operations and produces a [B, F] matrix of partial sums; the chips exchange and add their results. Let s be bytes per transmitted element: s = and Q = sBF bytes per chip. Sending and receiving happen simultaneously. Local HBM bandwidth is assumed not to limit computation.
If we assume we can perfectly overlap communication and computation, when Tmath > Tcomms, we see full utilization from our hardware. We call this being "compute-bound". When Tcomms > Tmath, we tend to be "communication-bound" and at least some fraction of our accelerator s/s is wasted waiting for data to be passed around.
If we optimize with the maximum in mind then the lower and upper bounds differ by at most a factor of 2 since Tmath + Tcomms ≤ 2 × max(Tmath, Tcomms). We then increase accuracy beyond this by modeling 'overlap regions' and overheads, which can be informed by profiling your specific model and target system.
✦ adaptation — Why the maximum, and why a factor of two?
If computation takes 5 ms and communication takes 3 ms, perfect overlap finishes in 5 ms; consecutive execution takes 8 ms. Each duration is at most their maximum, so their sum cannot exceed twice that maximum. These bounds use the selected rates; launch overhead, message latency, and data dependencies can make real execution slower.
One way to tell if an operation will be compute or communication-bound is to look at its "arithmetic intensity" or "operational intensity".
Definition: the arithmetic intensity of an algorithm is given by the ratio of the total s it performs to the number of bytes it needs to communicate — within a chipbetween chips. We write this as Intensity(Computation).
Arithmetic intensity measures the "s per byte" of a given operation. To a first order, when our arithmetic intensity is high, Tmath is large compared to Tcomms and we typically use most of the available s/s. When the opposite is true, we spend more time on comms and waste s. The point where this crossover happens is the "peak arithmetic intensity" of our hardware, the ratio of peak accelerator s/s to accelerator bandwidth. We write this as Intensity(Accelerator).
The quantity Intensity(Accelerator) is the arithmetic intensity at which our accelerator achieves its peak s/s. For , this is about /byte, since it can perform and load from HBMsend over the network. That means if an algorithm has a lower arithmetic intensity, it will be bound by byte loadingcommunication and thus we won't make good use of our hardware.
Example (dot product):
To compute the dot product of two vectors of length N = in bfloat16 precision, x · y: bf16[N], bf16[N] → bf16[1], we need to load x and y from memory, each of which has 2N bytes, perform N multiplications and N − 1 additions, and write 2 bytes back into HBM.
As N → ∞, the dot product has an arithmetic intensity of ½ or, put another way, the dot product does 0.5 floating point operations per byte loaded. This means our arithmetic intensity is lower than that of our hardware and we will be communication-bound.
✦ adaptation — Which compute units perform a dot product?
The relevant ceiling is the VPU's, rather than the MXU'sscalar/vector units', rather than the Tensor Cores'. The original TPU example makes the same distinction: TPU v5p's VPU has a critical intensity around 3 FLOP/byte, so the dot product is still communication-bound. The matmul plots use the matrix multiplication units' throughput.
Visualizing rooflines
We can visualize the tradeoff between memorycommunication and compute using a roofline plot, which plots the peak achievable s/s (throughput) of an algorithm on our hardware (the y-axis) against the arithmetic intensity of that algorithm (the x-axis). Here's an example log-log plot:
Above, as the intensity increases (moving left to right), we initially see a linear increase in the performance of our algorithm (in s/s) until we hit the critical arithmetic intensity of the hardware, at the selected bandwidth. Any algorithm with a lower intensity will be bandwidth (BW) bound and limited by the peak memorynetwork bandwidth. Any algorithm to the right will fully utilize our s/s. We can generally improve the performance of an algorithm either by increasing its arithmetic intensity or by increasing the memorynetwork bandwidth available.
✦ adaptation — Why does the sloping line turn into a flat roof?
At arithmetic intensity Intensity(Computation), each byte transferred corresponds to Intensity(Computation) operations. A bandwidth of W bytes/s therefore supports at most W · Intensity(Computation) operations/s: doubling intensity doubles this limit. Once W · Intensity(Computation) reaches C, compute throughput becomes the limit. More bandwidth cannot increase throughput beyond C operations/s. Setting W · Intensity(Computation) = C gives the critical arithmetic intensity at Intensity(Computation) = C/W.
✦ adaptation — Why does a bigger batch move the point right?
The same D × F weight matrix is reused for every token. Doubling B doubles the arithmetic but leaves the weight bytes unchanged. Only input and output traffic grows. When weight traffic dominates, this almost doubles operations per byte; at large B, growing activation traffic limits further improvement.
✦ adaptation — Why does B cancel, while D still matters?
Each chip sends B × F partial sums after doing B × D × F operations. The operation count and communication volume both scale linearly with B and F, so they cancel in BDF/(sBF). Increasing D instead lengthens each dot product without changing the output shape, giving more operations per transmitted element: Intensity(Computation) = D/s.
Matrix multiplication
Let's look at our soon-to-be favorite algorithm: matrix multiplication (aka matmul). Given two matrices X and Y, we can write X * Y → Z where X has shape bf16[B, D], Y has shape bf16[D, F], and Z has shape bf16[B, F].
✦ adaptation — Our FP8 matmul reads inputs and weights in FP8 and writes the output in BF16. NVIDIA Tensor Cores multiply the FP8 operands and accumulate in higher precision. Compared with BF16, input and weight storage halve, while the output byte count stays the same.
✦ adaptation — What does the FP8 estimate include?
E4M3 and E5M2 are eight-bit floating-point formats with different precision and range. Both use one byte per value. We count those values and the BF16 output, but omit scaling metadata, quantization work, and extra memory traffic. The selected rate is the GPU’s published dense FP8 Tensor Core throughput; the compute multiplier lets you estimate a lower achieved rate.
The network still sends partial sums in the communication data type you selected. Choosing FP8 computation does not automatically quantize those messages.
✦ adaptation — The quantization exercises assume one byte per INT8 element and ignore scales, conversion work, and padding. The all-INT8 exercise assumes twice the BF16 operation rate; this is an exercise assumption, not a vendor INT8 specification. Weight-only quantization keeps the BF16 compute rate.
For BF16 inputs and output, load 2BD + 2DF bytes, perform approximately 2BDF FLOPs, and write 2BF bytes back to HBM.The matmul count. The exact arithmetic count is BF(2D − 1): D multiplications and D − 1 additions for each of B × F outputs. We use the conventional 2BDF approximation, assume each input is fetched once, and cast the result to BF16 before writing it to HBM. Real tiled kernels may read data more than once.
✦ adaptation — Other precisions
For the selected precision, weights use w = bytes, inputs use a = bytes, and outputs use o = bytes. The same calculation becomes:
We can get a nice simplification if we assume our "batch size" B is small relative to D and F. Then we get Intensity(Computation) ≈ 2B/w, or Intensity(Computation) ≈ B for BF16.
This is a reasonable assumption for Transformer matmuls since we typically have a local (per-replica) token batch size B < 1024 (note, not sequences) but D and F > 8000. Thus we generally become compute-bound when our per-replica batch size is greater than tokens, a very simple rule!
✦ adaptation — When does the small-batch approximation break down?
The rule above assumes B is small relative to D and F, so weight transfers dominate HBM traffic. With Intensity(Computation) ≈ 2B/w, we need B ≳ w · Intensity(Accelerator) / 2 = tokens per batch. The takeaway above uses the full byte count for the selected dimensions.
The figures also count input reads and output writes, which use additional HBM bandwidth.
Solving the full inequality gives B ≥ w · Intensity(Accelerator) · DF / [2DF − Intensity(Accelerator) · (aD + oF)], provided the denominator is positive. At large B, activation traffic matters, so intensity saturates at 2DF/(aD + oF) = . If that is no greater than Intensity(Accelerator), no finite batch reaches the compute ceiling.
Actual kernels break these matrices into smaller tiles that fit on-chip memory and may read data more than once. Let bm be a tile’s number of input rows, bk its summed dimension, and bn its number of output columns. For BF16, ignoring output writes and reuse between tiles, intensity is approximately bmbn / (bm + bn). This whole-matrix roofline is optimistic; profiling or a tile-level byte count makes it more accurate.
To pick a somewhat contrived example, say we want to multiply two big matrices X ∼ [B, D] and Y ∼ [D, F] which are split evenly across 2 (along the D dimension). To do this multiplication, we can multiply half of each matrix on each chip (Z₀ = X[:, :D // 2] @ Y[:D // 2, :] and Z₁ = X[:, D // 2:] @ Y[D // 2:, :]) and then copy the resulting "partial sums" to the other chip and add them together. Say we can copy in each direction and perform on each chip. What are Tmath and Tcomms?
Tmath is clearly half of what it was before, since each chip is doing half the work. Now what about Tcomms? This now refers to the communication time between chips! This is just the total bytes sent divided by the network bandwidth.
✦ adaptation — Partial-sum precision and network assumptions
BF16 partial sums use two bytes per element. FP32 uses four; INT8 uses one. Weight storage does not change this communication byte count. Sending and receiving happen simultaneously, so bandwidth is measured in one direction. Reduction arithmetic, quantization cost, and message latency are omitted.
Note that, unlike before, the critical threshold now depends on D and not B! Try to think why that is. This is just one such example, but we highlight that this kind of roofline is critical to knowing when we can parallelize an operation across multiple .
✦ adaptation — What does this say about a larger cluster?
Only this two-chip exchange is modeled here. A larger collective has a different byte count, schedule, and topology. Recompute operations and traffic per participating chip, then compare with the matching bandwidth. Do not divide cluster-total FLOPs by one chip’s bandwidth, or multiply outgoing bandwidth by two because sends and receives overlap.
A full matmul followed by an exchange has no overlap. Overlapping computation and communication requires sending completed output tiles while computing others. Pipeline startup and completion, as well as data dependencies, can prevent some communication from overlapping computation. The overlap control illustrates the effect on execution time.
We’ll develop AllGather, ReduceScatter, and AllReduce costs for actual sharding schemes in How to Parallelize a Transformer for Training.
✦ adaptation — The hardware numbers
Every row uses dense BF16 throughput and . The peak arithmetic intensity Intensity(Accelerator) = C/W uses the published rates, before multipliers.
| Hardware | Dense BF16 / chip | Intensity(Accelerator) · FLOP/byte |
|---|
The ICI figure is the bandwidth of one link in one direction. The scale-out figure is a per-chip share of outgoing DCN bandwidth. The number and arrangement of links matter for larger collectives.
NVLink is aggregate outgoing bandwidth per GPU. Scale-out is a per-GPU share of NIC bandwidth. The GB200 400 Gb/s NIC share is a reference configuration; equal rates do not guarantee equal performance for different communication patterns.
Why doesn’t GB300 automatically move the BF16 roof? The GB200 and GB300 rack specifications both list 360 PFLOP/s of sparse BF16 across 72 GPUs: 360 / 72 / 2 = 2.5 PFLOP/s dense per GPU.
Both provide 8 TB/s HBM bandwidth per GPU, so this BF16 roof stays the same. GB300’s higher memory capacity and low-precision gains do not change the BF16 compute-to-HBM-bandwidth ratio.
Both provide 900 GB/s one-way NVLink bandwidth per GPU, so that network roof stays the same. GB300’s 800 Gb/s per-GPU scale-out connection does change the scale-out roof relative to the 400 Gb/s GB200 reference configuration.
NVIDIA’s Tensor Cores are its matrix multiplication units. We halve rates explicitly marked sparse; Rubin’s BF16 figure is already dense. NVLink send-plus-receive rates are divided by two. Gb/s NIC rates are divided by eight to get GB/s.
Vera Rubin: preliminary GPU peak specifications
NVIDIA’s linked datasheet lists 4 PFLOP/s dense BF16 per Rubin GPU. It marks these specifications as preliminary, “up to,” and subject to change. This is the peak used by this preset.
The same datasheet specifies 22 TB/s HBM4 bandwidth per GPU. The approximate BF16 batch threshold is therefore 4 PFLOP/s ÷ 22 TB/s ≈ 182 tokens; the full input/output traffic raises the exact threshold.
NVLink 6 provides 3.6 TB/s bidirectional per GPU, giving 1.8 TB/s outgoing. ConnectX-9 provides 1.6 Tb/s of scale-out connectivity per GPU, giving 200 GB/s outgoing. As with the other presets, the pairwise model assumes that bandwidth is usable by this exchange.
NVIDIA’s product webpage currently lists different bandwidth figures: 19.2 TB/s HBM4 per GPU. 3 TB/s NVLink and 0.45 TB/s bidirectional scale-out per GPU. This preset consistently uses the linked GPU datasheet and architecture articles. See the dated sources and conversion details.
A Few Problems to Work
These exercises use TPU v5e: 197 TFLOP/s BF16, 394 TOP/s INT8, and 820 GB/s HBM bandwidth. Each question specifies its own precision; B, D, and F follow the shared controls.
Question 1 [INT8 matmul]:
Say we want to do the matmul X[B, D] ·D Y[D, F] → Z[B, F] in int8 precision (1 byte per parameter) instead of bfloat16 (2 bytes per parameter) since TPUs can do matmuls faster in lower precision.
- How many bytes need to be loaded from memory? How many need to be written back to memory?
- How many total OPs are performed?
- What is the arithmetic intensity?
- What is a roofline estimate for Tmath and Tcomms? What are reasonable upper and lower bounds for the runtime of the whole operation?
Assume our HBM bandwidth is 820 GB/s and our int8 peak OPs/s is (about 2x bfloat16).
Click here for the answer.
- Because we're storing our parameters in int8, we have 1 byte per parameter, so we have BD + DF = loaded from HBM and BF = written back.
- This is the same as in bfloat16, but in theory int8 OPs/s should be faster. So this is still 2BDF = OPs.
- Arithmetic intensity is 2BDF / (BD + DF + BF) = . If we make the same assumption as above about B ≪ D and B ≪ F, we get an arithmetic intensity of 2B, meaning our rule becomes B > HBM int8 arithmetic intensity / 2. Using the numbers given, our rule is B > . Note that this is basically unchanged!
- Tmath = 2BDF / = and Tcomms = (BD + DF + BF) / W = , so a reasonable lower bound is max(Tmath, Tcomms) = and an upper bound is Tmath + Tcomms = .
Question 2 [INT8 weights + BF16 compute]:
In practice we often do different weight vs. activation quantization, so we might store our weights in very low precision but keep activations (and compute) in a higher precision. Say we want to quantize our weights in int8 but keep activations (and compute) in bfloat16. At what batch size do we become compute bound? Assume .
Click here for the answer.
Again assuming B is small, we have 2BDF bfloat16 FLOPs but only DF weight bytes (instead of 2DF in bfloat16). This means we become compute-bound when B > . This is a lot lower, meaning if we can do int8 weight quantization (which is fairly easy to do) but still do bfloat16 FLOPs, we get a meaningful win in efficiency.
Question 3 [exact byte counts]:
Taking the setup from Question 2, make a roofline plot of peak FLOPs/s vs. B for F = D = 4096 and F = D = 1024. Use the exact number of bytes loaded, not an approximation.
Click here for the answer and interactive plot.
Both shapes eventually achieve the peak hardware FLOPs/s, but the larger D and F achieve it sooner. D = F = 4096 reaches the ceiling at 137 tokens; D = F = 1024 needs 227 tokens.
Question 4 [Memory Rooflines for GPUs]:
Using the spec sheet provided by NVIDIA for the H100 SXM, calculate the batch size at which a bfloat16 matrix multiplication will become compute-bound. Tensor Core FLOPs numbers marked “with sparsity” are twice the dense value because they require structured sparsity.
Click here for the answer.
The reported BF16 throughput is 1,979 TFLOP/s with sparsity. The dense value is half that: 989.5 TFLOP/s. HBM bandwidth is 3.35 TB/s, so the approximate batch threshold is 989.5 / 3.35 ≈ 295 tokens. This weight-dominated approximation assumes B is small relative to D and F; include activation traffic for the exact threshold.
Next: How to Parallelize a Transformer for Training, or read How to Think About TPUsHow to Think About GPUs.