Caching

A cache is a fast, nearby copy of data that is otherwise expensive to get — expensive because it lives far away (across a network), because it has to be recomputed from scratch each time, or because the thing that holds it (a database, an external API) is slow or precious. The whole idea fits in one line: the cheapest query is the one you never make. You spend a little memory to buy back latency and to take load off something slower. Most performance wins in real systems are not clever algorithms; they are a cache placed exactly where the same answer is asked for over and over.

Why caching works: the latency gap

Caching pays off because the places data can live differ in speed by orders of magnitude. The numbers below are rough — they vary by hardware, load, and distance — but the ratios are what matter, and they are stable:

• Reading from local memory (RAM) in your own process: roughly 100 nanoseconds (0.0001 ms). This is a dictionary lookup — no network, no disk.
• Asking a cache server like Redis on the same network: roughly 0.2–1 millisecond. One network round-trip, no disk seek.
• Querying a database that has to touch disk and run a query plan: roughly 1–20 milliseconds for a simple indexed read, more if it is uncached or complex.
• A cross-region call (your service in Virginia, the data in Frankfurt): roughly tens of milliseconds (50–150 ms) just for the light to make the trip, before any work happens.

A cache server is maybe 10,000× slower than local memory but still 10–100× faster than a database round-trip, and it spares the database the work entirely. That gap is the entire economic case for caching: turn a 10 ms query that also burdens a shared database into a 0.3 ms lookup against a copy.

Where caches live

Caches sit at many layers, and a single request often passes through several. Each layer is a place to keep a copy closer to whoever is asking:

• CPU / in-process memory — a plain dictionary inside your running service (e.g. cache[user_id] = user). The fastest possible at ~100 ns, but it is private to one instance and vanishes on restart. Example: a web server keeping the last 1,000 rendered product pages in a local LRU map.
• A shared cache — a dedicated server like Redis or Memcached (in-memory key–value stores that every instance of your service can read from over the network). It survives a single service restart, is shared across the whole fleet so all instances see the same copy, and sits one network hop (~0.5 ms) away. Example: storing a user's session so any of 50 web servers can serve their next request.
• The database's own buffer cache — the database keeps recently-used data pages in RAM so a repeated query does not re-read the disk. This is a cache you get for free and often forget you already have. Example: the second time you run the same SELECT, it returns far faster because the rows are already in the DB's memory.
• The browser / HTTP cache — HTTP response headers (like Cache-Control: max-age=3600) tell the user's browser it may reuse a downloaded response for an hour and skip the network request entirely. Example: your site logo (logo.png) is fetched once and served from disk on every later page view.
• A CDN at the edge — a CDN (Content Delivery Network) is a fleet of servers spread around the world that hold copies of your content physically near users. A reader in Tokyo hits a Tokyo edge server instead of your origin in Virginia, turning a 150 ms trip into a 5 ms one and never bothering your origin at all. Example: Cloudflare or CloudFront serving your CSS, images, and cacheable API responses from the nearest city.

Read patterns: how data gets into the cache

The most common pattern is cache-aside (also called lazy loading): the application code itself manages the cache, checking it first and only touching the source of truth on a miss. The cache stays "off to the side" and knows nothing about the database. Here are the exact steps, with comments numbering each one:

def get_user(user_id):
    user = cache.get(user_id)          # 1. look in the cache first
    if user is None:                  # 2. a MISS -> the cache had nothing
        user = db.query(user_id)        # 3. read from the source of truth
        cache.set(user_id, user, ttl=300)  # 4. populate cache for next time (5 min)
    return user                       # 5. HIT on every later call within the TTL

Walk it through: the first call for a given user is a miss — the cache is empty, so the app pays the full database cost (~10 ms), then writes the result into the cache. Every subsequent call within the next 5 minutes is a hit — answered straight from the cache (~0.3 ms) without ever touching the database. The hit rate (fraction of reads served from cache) is the number that decides whether caching is worth it; a 95% hit rate means the database sees only 1 in 20 reads.

A close cousin is read-through: instead of the application checking and populating the cache by hand, the cache library itself sits in front of the database. The app always asks the cache; on a miss the cache transparently fetches from the database, stores the result, and returns it. Same effect as cache-aside, but the miss-handling logic lives in the cache layer rather than scattered through your code.

Write patterns: how updates flow back

Reads are only half the story. When data changes, you must decide how the write reaches both the cache and the durable source of truth.

• Write-through — every update writes to the cache and the database together, synchronously, before the write is acknowledged. The cache is always current, so reads are never stale. The cost: every write pays both the cache write and the (slower) database write, so writes are as slow as the database.
• Write-behind (also write-back) — the update writes to the cache now and returns immediately; the database is updated later, in the background, by a flush. Writes feel instant and you can batch many of them into one efficient database write. The cost: durability risk — if the cache dies after acknowledging a write but before the flush, that write is lost. You have traded safety for speed.

Expiry and eviction: nothing lives forever

Two different forces remove things from a cache, and it helps to keep them separate in your head.

TTL (time-to-live) is expiry on a timer. When you write an entry you stamp it with a lifespan — ttl=300 means "this entry is valid for 300 seconds, then it expires on its own." TTL is your main lever for bounding staleness: a 60-second TTL guarantees the cache is never more than a minute behind the source, no matter what else happens.

An eviction policy is what happens when the cache runs out of room — RAM is finite, and to admit a new entry the cache must throw out an old one. The policy decides which victim to drop:

• LRU (least-recently-used) — evict whatever has not been read or written for the longest time. This is the default in most caches, and the bet behind it is simple: data that was hot recently is likely to be hot again soon, so keep it and discard the cold stuff.
• LFU (least-frequently-used) — evict whatever has been accessed the fewest times overall, regardless of recency. Better when popularity is stable over time rather than bursty.
• FIFO (first-in, first-out) — evict the oldest-inserted entry, ignoring how often it is used. Simple but rarely ideal, since it can throw out a still-hot item just because it was inserted early.

TTL and eviction can both fire: an entry might be evicted (LRU) long before its TTL expires because the cache filled up, or it might expire (TTL) while there is still plenty of room. Either way the next read for that key is a miss and re-populates.

The hard parts

Everything above is the easy 80%. The genuinely hard part of caching is keeping copies correct under change and safe under load. There is an old joke that there are only two hard things in computer science: cache invalidation and naming things. The first one is the killer.

Cache invalidation: the staleness problem

Invalidation is the problem of stale data: the moment the source of truth changes, every cached copy is potentially wrong, and a cache has no way to know that on its own. A concrete scenario: a user changes their display name from "Sam" to "Samantha." The database row updates instantly, but the cached copy of their profile still says "Sam" — and every page that reads from the cache will show the old name until something fixes it. You have two basic tools, and both are imperfect:

• Expire on a timer (TTL) — let the stale entry die on its own after, say, 60 seconds. Dead simple, but it serves wrong data until the TTL lapses. For a display name, a minute of staleness is fine; for a price or a permission, it may not be.
• Explicitly delete on change — when the data changes, delete (or overwrite) the cached entry right then, so the next read re-populates from the fresh source. Always fresh, but it requires you to remember every place that fact is cached. Miss one — say the profile is cached under both user:42 and inside a cached "team roster" list — and you serve a wrong answer for hours until that other entry's TTL happens to lapse.

There is no clean, automatic answer, which is exactly why it is famous. The practical default is both: delete on change for promptness, plus a modest TTL as a safety net so any entry you forgot to invalidate still self-heals eventually.

Cache stampede: the thundering herd

The second hard part is a load problem. A cache stampede (or thundering herd) happens when a single hot key expires and, in that instant, every concurrent request misses the cache at once and slams the database together. Concrete scenario: the homepage feed is cached with a 60-second TTL and serves 5,000 requests per second. At the tick when it expires, all 5,000 in-flight requests find an empty cache, all 5,000 decide to rebuild it, and all 5,000 hit the database simultaneously — a query that one request could absorb is now multiplied 5,000-fold, sometimes hard enough to take the database down. Three common defenses:

• Locking / single-flight — when the key is missing, only the first request acquires a lock and rebuilds the entry; the other 4,999 wait briefly and then read the freshly-populated value. One database query instead of 5,000.
• Staggered / jittered TTLs — instead of setting every related key to expire at exactly the same time, add a small random offset (e.g. ttl = 300 + random(0..30)) so a batch of keys does not all expire on the same tick and stampede together.
• Serve-stale-while-refresh — when an entry is near expiry, keep serving the slightly-stale copy to everyone while a single background task refreshes it. No request ever sees a miss; readers tolerate a few seconds of staleness in exchange for never piling onto the database.

Note that a shared cache like Redis is itself a network node that can be slow, full, or down — see Distributed systems in practice for what happens when that single dependency fails and why you need a sane fallback to the source of truth.

The fundamental tradeoff

A worked example: caching a user profile

Let's trace cache-aside with a TTL end to end, through a miss, then a hit, then an update that invalidates. The data is a user profile keyed by user:42, cached for 300 seconds.

# --- Call 1: MISS (cache is empty) ---
get_profile(42)
  cache.get("user:42")        # -> None  (miss)
  row = db.query(42)         # ~10 ms: {"name": "Sam"}
  cache.set("user:42", row, ttl=300)
  return {"name": "Sam"}     # total ~10.3 ms

# --- Call 2: HIT (same key, seconds later) ---
get_profile(42)
  cache.get("user:42")        # -> {"name": "Sam"}  (hit!)
  return {"name": "Sam"}     # total ~0.3 ms, DB untouched

# --- Sam renames to "Samantha": UPDATE + INVALIDATE ---
update_profile(42, name="Samantha")
  db.update(42, name="Samantha")  # source of truth changes
  cache.delete("user:42")          # drop the now-stale copy

# --- Call 3: MISS again, re-populates with fresh data ---
get_profile(42)
  cache.get("user:42")        # -> None  (we just deleted it)
  row = db.query(42)         # {"name": "Samantha"}
  cache.set("user:42", row, ttl=300)
  return {"name": "Samantha"}

The key insight: had we forgotten the cache.delete("user:42") line on update, Call 3 would have been a hit returning the stale "Sam" for up to 300 more seconds — that is invalidation biting you in miniature. And the TTL is the safety net: even if we forget to delete, the wrong value self-corrects within 5 minutes. That combination — delete-on-change for promptness, TTL for safety — is the everyday workhorse pattern.

Related reading: Databases covers the buffer cache the DB already maintains for you and why a cache is a layer in front of it; Distributed systems in practice covers a shared cache as a failure-prone node; and Observability is where you watch your hit rate and catch a cache that has quietly stopped helping. For how caching fits into a larger architecture, see System design.