Caching in system design interviews: strategies, eviction and a worked example

In a system design interview, caching means keeping copies of frequently read data in a faster layer, usually memory, so most requests skip the slower database or service behind it. Interviewers expect you to say where the cache sits, how it is filled and updated, what gets evicted, how stale data is handled, and what happens when the cache fails or a single key gets very hot.

Why caching comes up in almost every design round

Most interview prompts are read-heavy: feeds, product pages, profiles, short links. When reads outnumber writes by a wide margin, the read path becomes the design, and a cache is the cheapest way to protect it.

Saying "add a cache" earns little. The questions that score are the next ones. What is the key? What is the TTL? What happens on a write? What happens when the cache restarts empty? The rest of this post is a checklist for those questions. If you want the full framework first, read system design interview questions.

Where caches live: CDN, application cache, database buffer

A request can hit several caches on its way to the data. Name the layer you mean.

• Client or browser cache: responses stored on the device, controlled by HTTP cache headers. Fast and free, but you cannot invalidate it from the server.
• CDN: edge servers close to users that cache static files and some API responses. Good for images, scripts and public pages. Weak for per-user or fast-changing data.
• Application cache: an in-memory key-value store, or an in-process cache, that your services read before the database. This is the one most interview answers are about.
• Database buffer cache: the database keeps hot pages in memory on its own. It helps, but it still costs a query, a connection and the database's CPU.

A common answer layers them: CDN for static content, an application cache for hot objects, and the database's buffer underneath.

Caching strategies interviewers ask about

Cache-aside (lazy loading)

The application checks the cache. On a miss, it reads the database, writes the result into the cache and returns it. On a write, it updates the database and deletes the cache entry. This is the most common default because the cache only holds data that was actually requested, and a cache outage degrades to slower reads rather than errors. The cost: the first read of each key is slow, and there is a window where stale data can appear.

Read-through

The same idea, but the cache layer itself loads from the database on a miss. Application code only talks to the cache. It is cleaner to use, and it ties you to a cache that knows how to reach your data.

Write-through

Every write goes to the cache and the database together, and the write completes when both succeed. Reads after writes are fresh. The cost is slower writes, and the cache fills with data that may never be read.

Write-back (write-behind)

Writes go to the cache and return immediately, and the cache flushes them to the database later, often in batches. Writes are fast and the database sees less load. The risk is data loss if the cache fails before it flushes, so it suits data you can afford to lose or rebuild, such as counters or view tallies, not payments.

Write-around

Writes go straight to the database and skip the cache. The cache only fills on reads. It avoids filling the cache with data written once and rarely read, like logs, at the price of a miss on the first read after a write.

Eviction policies: LRU, LFU and TTL

Memory is finite, so something has to leave. The common policies:

• LRU (least recently used): evict the key not touched for the longest time. A strong default for most access patterns.
• LFU (least frequently used): evict the key with the fewest accesses. Better when popularity is stable, worse when it shifts, because old favorites linger.
• TTL (time to live): every key expires after a set time, whatever its popularity. This is a freshness control more than a memory control, and it is usually combined with LRU or LFU.
• FIFO: evict the oldest inserted key. Simple, but it ignores how often a key is used.

How do you implement an LRU cache?

This also shows up as a coding question. The classic structure is a hash map for constant-time lookup plus a doubly linked list for recency. In Python, OrderedDict gives you both in one structure.

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity):
        self.capacity = capacity
        self.data = OrderedDict()

    def get(self, key):
        if key not in self.data:
            return None
        self.data.move_to_end(key)
        return self.data[key]

    def put(self, key, value):
        if key in self.data:
            self.data.move_to_end(key)
        self.data[key] = value
        if len(self.data) > self.capacity:
            self.data.popitem(last=False)

cache = LRUCache(2)
cache.put("a", 1)
cache.put("b", 2)
cache.get("a")        # "a" is now the most recent
cache.put("c", 3)     # evicts "b"
print(list(cache.data))  # ['a', 'c']

Every read moves the key to the end, so the front of the OrderedDict is always the least recently used. When a put pushes the size over capacity, popitem with last set to False removes the front. Both get and put are O(1). In an interview, say that a production version also needs a lock or a thread-safe structure if several threads share it.

Cache invalidation: keeping data fresh enough

Invalidation is where most caching follow-ups land. The practical options:

• TTL only: accept data up to N seconds old. Simple, and fine when the product tolerates it.
• Delete on write: after updating the database, delete the cache key so the next read reloads it. Deleting is usually safer than updating the cache, because two concurrent updates can otherwise leave the older value in the cache.
• Versioned keys: include a version in the key, such as product:42:v7, and bump the version on write. Old entries simply stop being read and age out.
• Event-driven: the database or service publishes change events, and a consumer invalidates the matching keys. More moving parts, but it works across many services.

Even with delete on write, a race remains: a reader can load the old value from the database just before the write and put it back into the cache just after the delete. A short TTL bounds how long that stale value lives. Saying this out loud shows you have operated a cache, not just drawn one.

Failure modes: stampedes and hot keys

Cache stampede and the thundering herd

When a popular key expires, or the cache restarts empty, many requests miss at once and all hit the database together. That is a stampede, sometimes called a thundering herd, and it can take the database down at exactly the moment the cache stops protecting it.

• Locks or single flight: the first request to miss takes a short lock and reloads the key. Others wait briefly or get the stale value.
• Request coalescing: inside one server, concurrent misses for the same key share one database call.
• Jittered TTLs: add a random offset to each TTL so keys written together do not all expire together.
• Early refresh: refresh a hot key shortly before it expires, or serve stale data while one background request refreshes it.
• Warm-up: preload the hottest keys before sending traffic to a new or restarted cache node.

Hot keys

A cache cluster spreads keys across nodes, but a single very popular key still lands on one node. A celebrity profile or a flash-sale product can overload that node while the rest sit idle. The usual answers: keep a small in-process cache on each application server for the hottest keys, replicate the hot key to several nodes and read from any of them, or split it into copies with a suffix such as item:42#3 and pick one at random. Each option trades extra staleness or memory for spread.

Consistency trade-offs

A cache is a second copy of the data, so you are always choosing how stale it may be. Decide per data type, and say it.

• Product descriptions or profile bios: seconds to minutes of staleness is usually fine.
• Inventory counts or prices at checkout: read from the source of truth at the moment it matters, even if the listing page used a cached value.
• A user's own edits: many products need read-your-writes, so after a user updates something, read their own data from the database or update their cache entry directly.
• Account balances and permissions: often not cached at all, or cached with very short TTLs and explicit invalidation.

Worked example: adding a cache to a read-heavy endpoint

Prompt: a product detail endpoint is slowing down at peak. These numbers are assumptions you would state in the round, not facts.

1. State the load. Peak traffic is 20,000 reads per second and about 200 writes per second, so reads outnumber writes 100 to 1. Assume the database handles around 5,000 reads per second comfortably. Without a cache, it is four times over budget.
2. Choose the pattern. Cache-aside in an in-memory key-value store, key product:{id}, value the serialized product. Delete the key on every product update.
3. Check the hit ratio. Product traffic is usually skewed toward a small set of items, so assume a hit ratio of 95%. Then the database sees 5% of 20,000, which is 1,000 reads per second, well inside its budget.
4. Size the memory. Say 2 million products are read often enough to matter, at about 2 KB each. That is about 4 GB, which fits on one cache node. Add a replica for failover rather than for capacity.
5. Estimate latency. Assume about 1 ms for a cache hit and 10 ms for a database read. A miss costs both, about 11 ms. The average is 0.95 times 1 plus 0.05 times 11, which is about 1.5 ms, down from about 10 ms.
6. Set freshness. TTL of five minutes with up to 30 seconds of random jitter, plus delete on write. Price and stock at checkout are read from the database, not the cache.
7. Name the failures. A popular product expiring causes a stampede, so use single flight per key. If the cache node fails, the replica takes over. If both fail, the database gets the full 20,000 reads per second, so add load shedding or a small in-process cache as a last line.

Spoken in the round, the summary might sound like this invented example:

Example: "Reads outnumber writes 100 to 1, so I'd put a cache-aside layer in front of the product table. At a 95% hit ratio the database drops from 20,000 to about 1,000 reads a second. I'll delete on write and keep a five-minute jittered TTL as a safety net, and checkout reads price from the database, since that's where staleness costs money."

The same structure fits other read paths. The redirect lookup in design a URL shortener, diagrammed is a good one to rehearse it on.

Common mistakes

1. Adding a cache with no reason. State the read-to-write ratio and the load first. A cache in a write-heavy system can add cost and bugs for little gain.
2. No invalidation story. If you cannot say what happens on a write, the interviewer will ask, and "it expires eventually" is rarely enough on its own.
3. Ignoring the cold start. An empty cache after a restart sends full traffic to the database. Mention warm-up or load shedding.
4. Caching what must be exact. Balances, permissions and checkout prices need the source of truth, or a very deliberate exception.
5. Treating the cache as durable. Unless you chose write-back on purpose and accepted the risk, the database is the source of truth and the cache can be lost at any time.

How to practice caching questions

1. Pick one read path. A feed, a profile, a product page or a redirect.
2. Say the numbers aloud. Read and write rates, hit ratio, memory size, latency before and after.
3. Run the checklist. Layer, pattern, key, eviction, TTL, invalidation, stampede, hot key, consistency.
4. Have someone push back. Or write three follow-ups yourself: what if the cache dies, what if one key gets very hot, what if a user sees their own stale edit.
5. Time the whole design. Caching is one deep dive inside a 45-minute round. Practice fitting it into about ten minutes.

Practice with the System design surface

HIDEA AI's System design surface returns an overview, a component list, a rendered architecture diagram and trade-offs. Use it to check a caching layer against a structure, then redraw the design and say the trade-offs in your own words. You speak or type every answer yourself.

It is one of four separate surfaces in the desktop app for macOS and Windows. Fifteen session minutes a day is free, with no credit card. See the interview copilot overview and the pricing page. If you plan to use any tool in a live round, read is using an AI interview copilot cheating and the employer's rules first.

Where to go next

• The full framework: system design interview questions.
• A complete design to practice on: design a URL shortener, diagrammed.
• Rehearse the clock: system design mock under a clock.
• Try a session: download HIDEA AI.