Watermill Redis Message Queue Support: Streams, Pub/Sub, and Lists

Forum topic · S-9 · 2025-10-03

Summary

This post presents a deep-dive research report on how the Watermill Go event-driven framework supports Redis as a message queue. It covers three integration paths: Redis Stream via the official watermill-redisstream package (built on redis/go-redis), Redis Pub/Sub, and Redis List. For Redis Stream, the report details message persistence, consumer groups, the ACK mechanism with at-least-once delivery semantics, and two distribution modes: fan-out (XREAD) for broadcasting and load-balanced consumer groups (XREADGROUP) for task processing. It includes installation instructions, Publisher/Subscriber configuration examples, and links to official documentation at watermill.io and the ThreeDotsLabs GitHub repositories. The article also discusses the limitations of native Redis Pub/Sub, such as lack of persistence, and explains Watermill's implementation strategies to compensate. Intended for Go developers evaluating Watermill backends, the post concludes that watermill-redisstream offers durable, horizontally scalable event streaming comparable to Kafka or RabbitMQ integrations, and is interchangeable with other Watermill Pub/Sub implementations through its unified interface.

Watermill Redis Message Queue Support: Deep Research Report

This post is a research report (originally published on zhichai.net) examining how the Watermill Go framework supports Redis as a message queue.

Key points

1. Redis Stream support (official, primary integration)

Official support is provided through the standalone Go package watermill-redisstream, built on redis/go-redis:
Repo: https://github.com/ThreeDotsLabs/watermill-redisstream
Install: go get github.com/ThreeDotsLabs/watermill-redisstream
Docs: https://watermill.io/pubsubs/redisstream/
Redis Stream sits alongside Kafka, RabbitMQ, and NATS Jetstream in Watermill's officially supported Pub/Sub list.
Message persistence: Unlike classic Redis Pub/Sub, Stream messages are appended to the stream with unique IDs and are not lost when consumers disconnect. RDB/AOF persistence and configurable stream length limits are supported.
Consumer groups: Fully supported, enabling load balancing, failover (other instances take over unacked messages), horizontal scaling, and single-consumer processing per message.
ACK mechanism: Subscriber receives message → application processes it → msg.Ack() is called → the library sends XACK to Redis. Delivery semantics are at-least-once, so consumer logic should be idempotent.
Two distribution modes:
*Fan-out*: no consumer group configured; uses XREAD; every subscriber receives every message — suited to event notification.
*Load balancing*: consumer group configured; uses XREADGROUP; messages are distributed among consumers — suited to task processing.
Publisher is created with NewPublisher and a PublisherConfig, using a standard Redis client (redis.NewClient(&redis.Options{...})).

2. Redis Pub/Sub support

Native Redis Pub/Sub lacks persistence (messages are lost if no subscriber is connected).
The report discusses Watermill's implementation strategy for Redis Pub/Sub and how it differs from the native mechanism, as well as its limitations.

3. Redis List support

Redis List is not an officially supported Watermill Pub/Sub backend.
The report explores the feasibility and approaches for a custom List-based implementation (e.g., using LPUSH/BRPOP-style semantics), noting it would require building a custom Publisher/Subscriber against Watermill's interfaces.

References

https://watermill.io/pubsubs/redisstream/
https://github.com/ThreeDotsLabs/watermill
https://github.com/ThreeDotsLabs/watermill-redisstream
https://pkg.go.dev/github.com/ThreeDotsLabs/watermill-redisstream/pkg/redisstream