System Design · System Design

Latency vs throughput

Time for one CampusClip open versus finishes per second, and Little's Law for in-flight work.

CampusClip's 0.2 s redirect and its 20 finishes/s are different kinds of number. This topic names them — latency and throughput — and shows they are not inverses once several opens overlap. Little's Law then turns the pair into a count of in-flight work you can budget a pool against.

System Design
Medium level
5 concepts

1Latency is time for one open

Latency is how long one CampusClip open takes, from the student's tap to the redirect reply. It is a duration, so its unit is seconds (or milliseconds). It is not how many students are opening links, and it is not how many redirects the host finishes in a second.

For this topic we give a measured mean latency: W = 0.2 s per redirect on host H when the host is keeping up. That 0.2 s is a classroom measurement we will reuse. It is not a claim about a "typical" shortener.

Figure. Latency W is the elapsed time of one CampusClip open, from the tap on clip.campus/a3k9 to the 302 on the wire. On host H that mean is the classroom measurement 0.2 s. It is a duration, not a finish count.

How to read one latency

StartStudent taps clip.campus/a3k9.
WorkCampusClip looks up a3k9 and builds the 302.
EndThe reply is on the wire. W is that elapsed time, averaged.

A CampusClip open takes 0.2 s from tap to redirect. That 0.2 s is

Latency — time for one request
Throughput — finishes per second
Load — arrivals per second

A duration on one request is latency. Finishes/s is throughput. Arrivals/s is load.

2Throughput is finishes per second

Throughput is how many CampusClip redirects finish in a second. It is a rate, so its unit is 1/s. On host H we already called the maximum finish rate capacity (20/s). Observed throughput is how many actually finished — it cannot exceed capacity, and it cannot exceed arrivals for long (you cannot finish work that did not arrive).

At 9:00, arrivals are 10/s and the host can finish 20/s, so observed throughput settles at 10/s — every arrival finishes, and the extra capacity sits as headroom. Throughput is not "the inverse of 0.2 s". That confusion is the next concept.

Figure. Steady state at 9:00: throughput matches arrivals (10/s). Capacity (20/s) is the ceiling, not the observed rate.

How to read throughput

Count finishesRedirects that completed in one second.
Cap itCannot exceed μ (20/s here) or, in steady state, λ.
Keep the unit1/s — not seconds, not students.

Arrivals 10/s, capacity 20/s, host keeping up. Steady-state throughput is

10 / s
20 / s
0.2 s

You finish what arrived. 20/s is unused capacity, not observed throughput. 0.2 s is latency.

3Latency and throughput are not inverses

If CampusClip handled one open at a time, throughput would be 1/W: one 0.2 s redirect would give 5 finishes/s. Host H does not work that way. It can have several opens in flight at once — overlapping lookups — so finishes per second can be larger than 1/W.

So you cannot read 20/s capacity as "each redirect takes 1/20 = 0.05 s", and you cannot read W = 0.2 s as "the host finishes 5/s". The missing piece is how many opens are in the host at once. Little's Law names that piece.

One-at-a-time versus overlapping
If the host…	Then throughput	CampusClip
Finishes one open before starting the next	1/W (here 5/s)	Not our host
Keeps several opens in flight	Can exceed 1/W, still ≤ μ	Host H, 20/s ceiling

Host H finishes 20 redirects/s with mean latency 0.2 s. Which claim is false?

Each redirect must take 1/20 = 0.05 s
Several opens can be in flight at once
Throughput 20/s and W = 0.2 s can hold together

The 0.05 s claim assumes one-at-a-time. Overlap lets 20/s and 0.2 s coexist; Little's Law will say that means 4 in flight.

4Little's Law and concurrency

Little's Law relates average in-flight work to arrival rate and latency: L = \lambda W. L is the average number of CampusClip opens on the host (or in the whole path you measured), \lambda is arrivals per second, and W is mean time an open spends in that same path. Units must match: if W is in seconds, \lambda is per second.

On host H at capacity, \lambda = 20/s and W = 0.2 s, so L = 20 \times 0.2 = 4 opens in flight. At 9:00, \lambda = 10/s gives L = 2. Rearranged, \lambda \approx L / W — that is throughput in terms of concurrency and latency, not a new law.

Figure. Little's Law is L = λW on one path. At host H capacity, λ = 20/s and W = 0.2 s, so L = 4 opens in flight. At 9:00, λ = 10/s gives L = 2. Units must match: seconds with per-second.

How to size in-flight work

MeasureTake steady-state arrival rate \lambda and mean latency W on the same path.
MultiplyL = \lambda W is the average number of concurrent opens.
BudgetThread pools, DB connections and queue depth must cover that L with headroom.

In-flight opens at capacity

CampusClip host H finishes 20 redirects/s. Mean latency is 0.2 s. How many opens are in flight on average?

\lambda = 20 / s, W = 0.2 sgiven
L = \lambda W = 20 \times 0.24 in-flight
Check: \lambda \approx L / W = 4 / 0.220 / s

Pro tip. If latency doubles at the same QPS, concurrency doubles with it. Convert milliseconds to seconds before multiplying.

At 50 opens/s with mean latency 200 ms, average in-flight opens are about

10
250
0.25

W = 0.2 s, so L = 50 \times 0.2 = 10. 250 multiplies by 5 s; 0.25 leaves W in milliseconds.

5Budget threads from L

Once L is known, it is a budget. If host H has 4 opens in flight on average at 20/s, a thread pool or connection pool sized at 2 will stall — the host is already holding more work than that. A pool sized at 4 is tight (no headroom). A pool sized at 8 covers L and a burst, using a number we chose, not an industry default.

The same identity warns you when latency grows. If W becomes 0.4 s at the same 20/s, L becomes 8. The pool that was "fine" at 4 is now the bottleneck you just created by letting latency double.

Figure. When W rises from 0.2 s to 0.4 s at the same 20/s, L goes from 4 to 8. A pool sized at 4 covered the old L and is now short by 4. Size the pool from the new L, not the old one.

How a pool size is checked

Compute LL = \lambda W on the path that uses the pool.
ComparePool size must be at least L, plus whatever headroom you choose to state.
Recompute when W movesIf latency doubles at the same λ, L doubles — the old pool is now short.

Latency doubles, pool stays

Host H still finishes 20/s but mean latency rises from 0.2 s to 0.4 s. What happens to in-flight work?

Was L = 20 \times 0.24
Now L = 20 \times 0.48
A pool of 4 versus new Lshort by 4

Pro tip. Same QPS, double latency, double concurrency. Size the pool from the new L, not from the old one.

λ = 20/s, W = 0.4 s. A connection pool of 4 is

Smaller than L = 8, so work will wait on a connection
Exactly L, so it is loose headroom
Larger than L, because 20/s needs 20 connections

L = 8. A pool of 4 is half of in-flight work. 20 connections would be one per QPS, which is the one-at-a-time mistake.

Notes

Latency W is time for one request (seconds). Throughput is finishes per second.
They are inverses only if the host is strictly one-at-a-time. Overlap breaks 1/W = QPS.
Little's Law: L = λW. L is average in-flight requests on the measured path.
Convert milliseconds to seconds before multiplying λ by W.
A pool smaller than L becomes a wait; if W doubles at the same λ, L doubles.

Formulas

Little's Law: L = \lambda W.
Throughput \approx L / W (rearrangement, same path).
One-at-a-time only: throughput = 1/W.

Exam traps & shortcuts

If someone says "20 QPS means 50 ms latency", they assumed one-at-a-time. Ask for L.
200 ms is 0.2 s. Leaving W in milliseconds makes L a hundred times too big.

Reference tables

Same CampusClip host H. Do not mix units across a row.

Latency, throughput, L
Symbol	Meaning	Classroom value
W	Mean time for one open	0.2 s
λ or QPS	Arrivals / finishes per second	10/s at 9:00; 20/s at capacity
L	Mean in-flight opens	2 at 9:00; 4 at capacity

Recap

Duration, rate, and the product that is concurrency.

Latency: Time for one open. Unit: seconds. Here W = 0.2 s.
Throughput: Finishes/s. Cannot exceed min(λ, μ) in steady state.
Not inverses: 1/W is throughput only if the host is one-at-a-time.
Little's Law: L = λW. 20/s × 0.2 s = 4 in flight. Convert ms → s first.
Pools: Size from L. If W doubles at the same λ, L doubles.

Practise Latency vs throughput

Reading is free and needs no account. Practice, mocks and progress live in the app.

A 3-question practice set that ends the chapter
5 quick checks with worked explanations
Timed mocks scored with the real marking scheme
Readiness tracked per topic, kept on your device

Continue with Google — freeNo card, no trial. Works offline once installed.