Horizontally-scaled LLM streaming

The problem

Stream LLM output to thousands of users across their devices without dropping tokens when a socket or a pod dies mid-response.

Approach

Socket.IO + Redis pub/sub sustaining 5,000 concurrent streams at p99 <20ms delivery. Custom resumable streams replay from the last delivered sequence — zero data loss and seamless handoff across a user’s devices.

socket.on('resume', async ({ streamId, lastSeq }) => {
  for await (const chunk of store.replay(streamId, lastSeq)) {
    socket.emit('token', chunk)
  }
})