Weik.io

Streams

Manage durable event streams with NATS JetStream

Streams keep a durable, ordered log of events so data is not lost when a consumer goes offline. Weik.io uses NATS JetStream to provide this capability.

What are streams?

A stream is an append-only log. Unlike a traditional queue such as RabbitMQ, where a message is removed once it is consumed, a stream retains CloudEvents. If an API endpoint you push to goes down for maintenance, the events stay in the stream, and you can replay the missed events when the endpoint recovers.

Why use streams?

  • Disk storage keeps events safe if a server restarts.
  • Strict ordering means events process in the exact sequence they arrived.
  • You can attach multiple consumers to the same stream without them interfering with each other.

How streams are created

You usually do not create streams manually. Weik.io creates them automatically:

Event Source streams

When you configure an Event Source, Weik.io automatically creates a stream named eventsource-{name}-stream. It captures everything that specific source brings in.

Event Channel streams

When you set up an Event Channel, you get an eventchannel-{name}-stream. This buffers the outgoing CloudEvents before delivery.

Configuration

Retention policies control how much data a stream keeps and prevent unbounded disk usage:

  • Age: Keep messages for a set period, then drop them.
  • Limits: Only keep the latest 10,000 events.
  • Size: Stop storing when the stream hits 5GB.
  • Interest: Keep events only until every active consumer has seen them.

You can also choose between file storage (durable) and memory storage (faster, but lost on restart).

Monitoring

The UI provides a dashboard for monitoring your streams. Monitor:

  • Total message count and disk usage.
  • The gap between the first and last sequence numbers (to see how far back your retention goes).
  • The number of active consumers. A drop to zero indicates that a consumer has stopped.

Consumers

Consumers are the processes that read from a stream. They can pull data at their own pace, or the server can push data to them. If a durable consumer crashes, NATS retains its position so it can resume exactly where it stopped.

When streams matter

Streams typically run in the background without intervention. They become relevant when:

  • You need to run an audit and see exactly what an external system sent you in the past.
  • A downstream service failed silently, and you need to replay the last 500 events.
  • You want to connect a second system to the same API feed without changing the original integration.

Recommendations

  • Set a retention limit. If you do not need to replay data older than 7 days, set a 7-day age limit to control disk usage.
  • Default to file storage. Memory storage is faster, but in-flight data is lost when a container restarts.
  • Monitor consumer lag. If the stream grows but the consumer does not advance, the target system is likely backed up or returning errors.

Next steps