We had a real-time game backend—2 auth services and 15 game services (the same system we discussed in a previous post), each with their own PostgreSQL instance, all communicating over HTTP with a shared Redis instance handling session data. Under normal traffic, everything worked fine. Redis wasn’t overwhelmed, the auth services were keeping up, and the load balancer was doing its job across replicas.
Then we load tested at 10,000 concurrent users. The Redis instance started hitting its hosted connection limit. When that happened, every game service fell back to calling the auth service directly over HTTP. That fallback turned a Redis bottleneck into an auth service bottleneck—and the auth service, now receiving requests from all 15 game services simultaneously on top of its own load, eventually ran out of RAM and crashed. Most requests started returning 401s.
This is the story of how we diagnosed it and fixed it with a cache hierarchy that dropped the load on both Redis and the auth services to a fraction of what it was before.
Table of Contents
The System Setup
To understand why the bottleneck happened, it helps to understand how auth ownership was structured.
We had two auth services—one responsible for the user auth flow and one for the admin auth flow. Each had its own completely separate PostgreSQL database, its own SQS queues for task delegation, and its own codebase. The game services had no direct access to either database. If they needed to know anything about a user or admin, they had to go through the auth services or through Redis.
The shared Redis instance was the exception to this isolation—it held session data and basic user information like id, email, and username that multiple services needed to read. Critically, though, only the auth services could write to Redis. The game services had read-only access. When a user logged in, the auth service created the session in Redis. When they logged out, the auth service deleted it. The game services just read whatever was there.
This ownership boundary is a deliberate microservices pattern and worth explaining briefly: when multiple services can write to the same data store, you end up with implicit coupling—one service’s bug can corrupt data that another service depends on, and debugging that across service boundaries is a nightmare. By giving Redis write ownership exclusively to the auth services, we ensured that session state had a single source of truth and a single team responsible for it. The game services were consumers, not owners.
The auth flow on every incoming request to a game service looked like this:
- Decode the JWT locally to check if it had expired—no network call, just CPU
- Check Redis for the session to confirm it was still active (handles logout)
- If Redis was unreachable or returned nothing, fall back to an HTTP call to the relevant auth service
The Bottleneck
This setup had a hidden assumption baked into it: that Redis would always be available and fast. Under normal traffic, that assumption held. But at 10,000 concurrent users, it broke in a cascade.
The first thing to go was Redis. Our hosted Redis instance had a connection limit enforced at the infrastructure level, and 15 services each opening multiple connections under heavy load pushed it over that limit. New connection attempts started getting refused.
When a game service couldn’t reach Redis, the code fell back to calling the auth service over HTTP. That fallback was designed as a safety net for the rare case where Redis went down briefly. It was not designed to handle 15 services simultaneously failing over at full load.
So now the auth service was handling its own 10,000 requests plus fallback requests from every game service that couldn’t reach Redis. The request queue grew faster than it could drain. Memory filled up. The auth service crashed. With the auth service down and Redis at its connection limit, the only response the system could give most requests was a 401. From the user’s perspective, the entire platform had gone down.
The root cause was that we had a single point of failure in the auth path—every request, for every user, on every service, was going through the same Redis instance and the same auth service with no local buffering whatsoever.
Introducing the In-Memory Cache
When we looked at what was actually being fetched from Redis and the auth service, the answer was almost embarrassingly simple: user id, email, and username—data that barely ever changes. The Redis check existed not because the data was dynamic, but because it was the mechanism for detecting logouts. If a session was deleted from Redis, the next check would miss, and the game service would know the token was no longer valid.
The insight was that we could decouple those two concerns. We could cache the user data locally in each service instance, and handle logout propagation separately through a different mechanism. That’s what we did.
We introduced an in-memory LRU cache using the lru-cache package. The idea is straightforward: the first time a token comes in and passes the Redis or HTTP check, we store the result locally. Every subsequent request with that same token skips Redis and the auth service entirely and just reads from memory. For a typical game session of 10-20 minutes, a user might make hundreds of requests—under the old system, each one hit Redis. Under the new system, only the first one does.
The updated request flow became:
- Decode JWT locally to check expiry
- Check the local LRU cache—if found, use it and continue
- If not found, check Redis
- If Redis is unreachable, fall back to the auth service HTTP call
- Store the result in the local cache for future requests
The cache implementation had a few details worth calling out:
First, each cache entry TTL is bounded by the JWT’s own expiry time. When we store a result, we calculate how much time is left on the token and use that as the upper bound for the cache TTL:
function calculateTtl(tokenExpirySec: number): number {
const remainingMs = tokenExpirySec * 1000 - Date.now();
const maxTtlMs = MAX_CACHE_TTL_MS;
return Math.max(1, Math.min(remainingMs, maxTtlMs));
}
This means cache entries can never outlive the token itself. Even if a logout event somehow never reached a service instance, the cache entry would still expire when the token does. It’s a secondary safety net that costs nothing.
Second, we added jitter to the TTL:
const jitter = Math.random() * 0.4 + 0.8; // multiplier between 0.8x and 1.2x
const ttl = Math.min(remainingMs, maxTtlMs * jitter);Jitter might seem like a minor detail, but it matters at scale. Without it, all 15 service instances that cached the same token at roughly the same time would expire it at exactly the same time. When that token’s next request comes in, all 15 instances would simultaneously miss the cache and hit Redis—a thundering herd. Randomizing the TTL by ±20% spreads those expirations out so they don’t all fire at once.
Third, the cache is keyed by the raw Bearer token and stores both user and admin entries in a single cache instance, using a discriminated union to keep them type-safe:
type CachedAuth =
| { kind: 'user'; data: UserData }
| { kind: 'admin'; data: AdminData };
const cache = new LRUCache<string, CachedAuth>({
max: MAX_CACHE_ENTRIES,
ttlAutopurge: true,
});The kind field is what makes this safe. When you retrieve an entry, you check kind before accessing data so TypeScript knows exactly what shape the data is. One cache instance handles both auth flows without any risk of a user entry being misread as an admin entry or vice versa. It also means only one cache needs to be managed, monitored, and invalidated—not two separate ones.
Logout Propagation
The local cache introduced an obvious problem: when a user logs out, the auth service deletes their session from Redis. But every service instance that has cached that token locally still has it. Without some form of invalidation, that user’s token would remain valid on those instances until the cache TTL expired naturally.
We already had SQS queues set up for task delegation between services, so the solution fit naturally into the existing infrastructure. When the auth service processes a logout request, it does two things: it deletes the session from Redis as before, and it publishes a logout event to an SNS topic.
It’s worth clarifying the SNS/SQS split here because it trips people up. SNS is a broadcast mechanism—you publish one message, and it fans out to every subscriber. SQS is a queue—each service instance pulls from its own queue at its own pace. Each game service has its own SQS queue subscribed to the SNS topic. So when the auth service publishes one logout event to SNS, every game service’s SQS queue receives a copy of that message. This is the standard AWS fan-out pattern.
// Auth service — on logout
await redisClient.del(`session:${token}`);
await snsClient.publish({
TopicArn: SESSION_INVALIDATION_TOPIC,
Message: JSON.stringify({ token }),
});
// Each game service — SQS consumer
async function handleLogoutEvent(message: { token: string }) {
localCache.delete(message.token);
}When the game service processes the SQS message, it just deletes the token from its local LRU cache. The next request with that token will miss the cache, hit Redis, find no session there, and return a 401. The token is effectively invalidated across the entire system.
The honest tradeoff here is that SQS delivery is not instantaneous. There is a small window — usually under a few seconds depending on queue pressure — between when the logout happens and when every service instance processes the event and clears its cache. During that window, a logged-out token could technically still be accepted by a service instance that hasn’t processed the event yet.
For our use case, this was an acceptable tradeoff. The window is short, the JWT expiry is a hard backstop, and the scenarios where a few extra seconds of session validity would cause real harm were not relevant to a gaming platform. If your requirements are stricter—financial transactions, healthcare, anything where session termination needs to be truly instant—this approach needs adjustment. You might skip the local cache for those specific endpoints, or use a much more aggressive TTL.
The final auth flow looked like this:
Results
The impact was immediate. The load on the auth services dropped dramatically—instead of receiving a request on every authenticated call from every service, they only received requests on actual cache misses, which in practice meant once per user per service instance per session.
Redis connection pressure dropped by the same proportion. The connection limit that was being blown through under load was no longer being approached because the vast majority of session lookups never reached Redis at all.
Latency across all services improved noticeably. An LRU cache lookup is a hashtable read in local memory—microseconds. A Redis call, even a fast one, involves a network round trip. At high concurrency, eliminating that round trip from the hot path of every single authenticated request added up quickly.
The cascade failure mode that took down the auth service was also eliminated. Even if Redis hit its connection limit again, the game services would mostly be serving from the local cache rather than all falling back to the auth service simultaneously.
Key Takeaways
- Session and user identity data is read-heavy and write-light—it’s one of the best candidates for caching. If the same data is being fetched from a remote source on every request and it almost never changes, local caching will almost always be worth it.
- Build cache hierarchies from cheapest to most expensive. In-process memory is faster than Redis; Redis is faster than an HTTP call. Exhaust cheaper options first before going to the network.
- Every caching layer introduces a consistency window. Be explicit about the tradeoff and make sure it is acceptable for your specific use case. The right answer is not always the same.
- TTL jitter is not optional at scale. If all your service instances expire the same cache entries at the same time, you replace a steady load with periodic spikes. Randomize your TTLs.
- Bounding cache TTL by token expiry gives you a free safety net. Cache entries that cannot outlive their tokens will self-clean even if invalidation events are missed.
- Event-driven invalidation via SNS fan-out to per-service SQS queues is the right pattern for propagating cache invalidation across service instances in a microservices architecture.
We went from hitting Redis or the auth service on every single authenticated request across 15 services to hitting them only on cold cache misses. At 10,000 concurrent users, that is the difference between a system that scales and one that crashes.