Weik.io

Streams

Manage durable event streams with NATS JetStream

Message queues are great until a consumer drops offline and you lose half a day of data. Streams solve this by keeping a durable, ordered log of everything that happens. In Weik.io, we use NATS JetStream to handle this.

What are Streams?

Think of a stream as an append-only log. Unlike a traditional queue like RabbitMQ where a message vanishes the second it’s read, streams hold onto CloudEvents. This means if an API endpoint you’re pushing to goes down for maintenance, the stream just waits. When the endpoint comes back, you can replay the missed events.

Why use Streams?

  • Disk storage keeps events safe if a server restarts.
  • Strict ordering means events process in the exact sequence they arrived.
  • You can attach multiple consumers to the same stream without them interfering with each other.

How We Create Streams

You don’t usually have to build these manually. The system handles it:

Event Source Streams

When you configure an Event Source, Weik.io automatically creates a stream named eventsource-{name}-stream. It captures everything that specific source brings in.

Event Channel Streams

When you set up an Event Channel, you get an eventchannel-{name}-stream. This buffers the outgoing CloudEvents before delivery.

Configuration

Streams will eat your entire disk if you don’t tell them when to stop. You can control this through retention policies:

  • Age: Keep messages for a few days, then drop them.
  • Limits: Only keep the latest 10,000 events.
  • Size: Stop storing when the stream hits 5GB.
  • Interest: Keep events only until every active consumer has seen them.

You can also choose between writing to disk (safer) or keeping everything in memory (faster but gone on restart).

Monitoring

The UI gives you a dashboard to watch your streams. You probably only need to look at this when something breaks. Keep an eye on:

  • Total message count and disk usage.
  • The gap between the first and last sequence numbers (to see how far back your retention goes).
  • The number of active consumers. If this drops to zero, something crashed.

Consumers

Consumers are just the processes reading from the stream. They can pull data at their own pace, or you can have the server push data to them. If a consumer crashes, NATS remembers where it left off (if it’s a durable consumer) so it can resume exactly where it failed.

When to Care About Streams

Most of the time, streams just run quietly in the background. But they matter when:

  • You need to run an audit and see exactly what an external system sent you last week.
  • A downstream service failed silently, and you need to replay the last 500 events.
  • You want to wire up a second system to listen to the same API feed without touching the original integration.

Practical Advice

  • Don’t keep data forever. Disk space costs money. If you don’t need to replay data older than 7 days, set a 7-day age limit.
  • Default to file storage. Memory storage is fast, but explaining to your team why an in-flight transaction disappeared during a container restart is not fun.
  • Watch your consumer lag. If your stream is growing but the consumer isn’t advancing, your target system is likely backed up or throwing errors.

What’s Next