The single ConnectionMultiplexer that wasn't the problem

The reads were perfect right up until they weren’t. In a steady-state load test, our Redis cache served GETs in one to five milliseconds — exactly what a cache is for. Then we turned the dial up to production-grade traffic: randomised calls, an aggressive cache-aside layer so repeated segments came from Redis instead of SQL, the works. And the same reads that had been sub-five-millisecond started stalling for 800 milliseconds to three seconds.

Not all of them. Not predictably. Just enough of the P95 to make the whole thing feel haunted. A cache that’s occasionally slower than the database it’s protecting is worse than no cache at all, because now you’re paying for the miss and the lookup. So we did what everyone does, and went looking for the guilty connection.

Why you blame the multiplexer first

If you’ve used StackExchange.Redis, you know the first rule: you don’t open a connection per operation. You create one ConnectionMultiplexer and you share it. It’s expensive to establish, it’s thread-safe by design, and it pipelines everything over a single connection. The docs are emphatic about this. So is every blog post. So is the colleague reviewing your PR.

Which means the moment Redis looks slow under load, the single shared multiplexer is the obvious suspect. It’s the one thing you were told to share, it’s right there in the hot path, and “the connection can’t handle the concurrency” is a story that writes itself. Surely it’s a pool problem. Surely one connection, however clever, can’t fan out across this much traffic.

So we split it. One multiplexer for writes, one for reads, on the theory that isolating the read path would let cached values flow independently of write contention. It was a reasonable change. It even helped a little. But it was a fix aimed at the wrong organ — we were tuning the plumbing because we’d already decided the plumbing was the problem.

The two configs we never read

Here’s the part that still stings a little. Before any of the clever diagnosis, the biggest win came from admitting we hadn’t read the documentation.

The first miss was server-side. Redis has had multi-threaded I/O since 6.0, and it is off by default — a single thread handles socket reads and writes unless you opt in. Under the concurrency we were now throwing at it, that single I/O thread was the ceiling. Two lines in redis.conf we’d simply never added:

# Multi-threaded I/O is opt-in. Default is one thread for all socket I/O.
io-threads 4               # roughly your core count, don't get greedy
io-threads-do-reads yes    # actually thread the read path, not just writes

The second miss was client-side, and worse, because it meant we’d been wasting hardware we were already paying for. We ran replicas. We had a whole Sentinel-managed topology of them. And our code was sending every single read to the master, because StackExchange.Redis has no global “read from replicas” switch — you ask for it per command, or you don’t get it:

// Reads: prefer a replica, fall back to the master only if none are available.
await db.StringGetAsync(key, flags: CommandFlags.PreferReplica);

// Writes: never let these drift onto a replica.
await db.StringSetAsync(key, value, flags: CommandFlags.DemandMaster);

One flag. PreferReplica on the reads, DemandMaster to keep the writes honest. That one change — finally using the replicas instead of hammering the primary — gave us a clean 5x improvement on read throughput under load.

Five times faster, from reading the manual. Nothing was wrong with Redis. Redis was sitting there, fully capable, waiting for us to configure the read path it had offered all along. It’s a humbling kind of bug: the system wasn’t broken, we just hadn’t finished setting it up.

The stalls that survived

The 5x was real, but it didn’t kill the haunting. The averages looked great; the P95 still spiked into the hundreds of milliseconds at peak. Something was holding individual reads hostage in a way that more throughput couldn’t fix — and that’s the tell. When your median is fine but your tail is on fire, you’re not CPU-bound or network-bound. You’re waiting on something.

The something was the .NET thread pool. StackExchange.Redis does its socket I/O asynchronously under the hood, and under heavy concurrent load the global ThreadPool was getting starved — the classic sync-over-async squeeze, worth its own article — every available worker thread blocked, nothing left to service the continuation that would hand your cached value back. The read had already returned from Redis in a millisecond. It was your own process that couldn’t find a thread to deliver it on. The “Redis latency” was, embarrassingly, a queue inside our own application.

This is the part the title has been circling. The connection multiplexer was never the problem. The connection was never the problem. The problem was that Redis socket processing and the rest of our workload were fighting over the same thread pool, and under load, the workload won.

The fix is to stop sharing. StackExchange.Redis lets you hand a ConnectionMultiplexer its own SocketManager with a dedicated set of I/O threads, isolated from the global pool:

// One dedicated SocketManager, shared by both multiplexers.
// Its threads service Redis socket I/O OFF the global .NET ThreadPool,
// so a starved ThreadPool can't stall reads that already came back.
_socketManager = new SocketManager("RedisIO", workerCount: 40);

var options = ConfigurationOptions.Parse(connectionString);
options.SocketManager = _socketManager;        // pass it in via options
return ConnectionMultiplexer.Connect(options);

Forty dedicated I/O threads, living entirely outside the pool that the rest of the app was busy starving. The default shared SocketManager uses around ten threads and leans on the global pool in places; there’s even a SocketManager.ThreadPool option, and it is pointedly not the default — precisely because it’s the thing that starves under exactly this load.

One disposal gotcha worth the comment, because it bit us later: a multiplexer does not own its SocketManager. Disposing the connection won’t dispose the manager. You created it, so you clean it up — last, after the connections:

// Multiplexer.Dispose() does NOT dispose the SocketManager — ownership is yours.
_writeConnection?.Dispose();
_readConnection?.Dispose();
_socketManager?.Dispose();   // dispose it yourself, after the connections

With Redis I/O on its own threads, the tail flattened. Across the entire production-grade run, reads sat at a stable one to five milliseconds — the numbers we’d seen at low load, now holding all the way up. No more three-second ghosts.

Is it Redis, or is it you?

If a fast cache starts stalling under load, here’s the order I’d check things now — cheapest and most likely-to-be-your-fault first:

Did you configure the read path at all? PreferReplica on reads, DemandMaster on writes, and io-threads-do-reads yes server-side. The biggest wins were the ones we’d never switched on.
Is your median fine but your tail awful? That’s thread-pool starvation, not Redis. Give Redis its own SocketManager so its I/O can’t be held hostage by the rest of your app.
Only then, look at the connection. Splitting read/write multiplexers is a real optimisation, but it’s a tuning knob, not a cure. If you reach for it first, you’re treating a symptom you haven’t diagnosed.
Measure before and after each change, separately. We’d have learned all of this faster if we hadn’t bundled fixes. The 5x and the tail-flattening were different problems with different causes; lumping them together nearly hid both.

What it was actually about

We went in certain that one shared connection couldn’t keep up, and we came out having barely touched it. The multiplexer was fine. Redis was fine. What was missing was a server config we’d never read, a client flag we’d never set, and an appreciation that a cache library doing async socket I/O can be brought to its knees by a thread pool that has nothing to do with Redis at all.

The honest lesson isn’t a config snippet. It’s that “it can’t be us, look how much load Redis is under” is the exact sentence that keeps it being us. The database was doing its job the whole time. We just hadn’t read far enough into the manual to let it.

The multiplexer never had a bad day. We did, for about a week, and then we read the docs.