Really Small Message Broker: Low-Overhead Event StreamingIn a world where distributed systems are becoming the default architecture for scalable applications, messaging infrastructure plays a critical role. Yet many teams find traditional message brokers—feature-rich systems like Kafka, RabbitMQ, and Pulsar—overkill for small projects, resource-constrained environments, or edge/IoT deployments. This article explores the concept, design, and practical uses of a “Really Small Message Broker” (RSMB) centered on low-overhead event streaming. You’ll learn why minimalism can be an advantage, key design principles, trade-offs, example architectures, implementation patterns, and when to choose an RSMB instead of a heavyweight broker.
Why “Really Small” matters
Low resource usage. A small broker consumes minimal memory, CPU, and disk, making it suitable for containers, single-board computers (Raspberry Pi), and serverless functions with tight limits.
Simplicity of operation. Fewer moving parts mean easier deployment, monitoring, and debugging. For teams without dedicated ops, a lightweight solution reduces operational overhead.
Faster time-to-value. For greenfield projects or prototypes, you can get messaging up quickly without deep configuration, schema registries, or complex client ecosystems.
Lower latency for simple flows. If your requirements are mostly pub/sub and ephemeral messages, a streamlined broker can outperform heavy systems by cutting protocol, persistence, and acknowledgement complexity.
Typical use cases
- Edge computing and IoT: devices with limited compute and intermittent connectivity.
- Microservices in small clusters: when intra-cluster messaging is simple and bounded.
- Prototyping and startups: moving fast without investing in infrastructure.
- Local development: replicate messaging behavior without spinning up heavy services.
- Resource-cost-sensitive applications: cloud functions or low-tier VMs where every MB and CPU cycle counts.
Core design goals
- Minimal runtime footprint: small binary, lightweight dependencies, compact memory usage.
- Simple API surface: a tiny set of operations (publish, subscribe, ack, optionally replay).
- Optional persistence with small disk footprint: allow ephemeral in-memory mode and compact on-disk storage when needed.
- Predictable behavior: clear semantics for delivery (at-most-once, at-least-once, or best-effort), timeouts, and backpressure.
- Interoperability: support common transport (HTTP/REST, WebSocket, or a tiny TCP protocol) and client libraries or simple language-agnostic formats (JSON, Protobuf).
- Security basics: TLS for transport and simple authentication (API key, token).
Minimal feature set (what to include)
- Pub/Sub primitives: topics, publishers, subscribers
- Lightweight subscription semantics: push (broker -> client) or pull (client polls)
- Simple QoS modes: best-effort (no ack), at-least-once (ack required), optional at-most-once (no retries)
- Message TTL and retention policy (configurable per topic)
- Optional persistence: append-only log with segment rotation and compaction toggle
- Small admin interface: topic creation, metrics, and health check endpoints
- Backpressure and rate limits to avoid memory blowouts
What to leave out (and why)
- Distributed consensus and partition rebalancing (no heavy leader election layers)
- Complex transactions across topics
- Full-blown schema registries and pluggable connectors
- Enterprise features like role-based access control (beyond basic auth), multi-tenant isolation, or extensive monitoring integrations
Excluding these reduces complexity and keeps the broker truly small and maintainable. If you later need them, you can evolve or integrate with specialized systems.
Architecture patterns
Below are a few practical architectures where an RSMB fits well.
- Single-process broker
- Use case: local development, single-node production for small fleets.
- Components: a single process exposes HTTP/WebSocket or a small TCP protocol, holds in-memory queues, optionally persists to local disk.
- Pros: minimal latency and simplest deployment.
- Cons: no high availability.
- Broker with lightweight persistence
- Use case: edge devices that need durability when connectivity drops.
- Components: in-memory queue + local append-only log. On restart, replay unacknowledged messages.
- Pros: durability without distributed complexity.
- Cons: storage limited by device.
- Broker behind a load balancer (stateless frontends + small stateful nodes)
- Use case: small clusters where horizontal scaling is needed but full distribution complexity is undesired.
- Components: stateless API frontends route to small stateful brokers by topic affinity (consistent hashing or sticky sessions).
- Pros: simpler scaling than full partitioning.
- Cons: increased operational coordination.
Protocol and API choices
Keep APIs simple and language-agnostic to maximize adoption. Common options:
- HTTP/REST: publish via POST /topics/{name}/publish; subscribe via long-polling GET /topics/{name}/poll or WebSocket upgrades for push.
- WebSocket: real-time streaming with simple subscribe/unsubscribe messages.
- Compact TCP protocol: binary framing with small headers for embedded clients.
Message payloads: JSON for simplicity, Protobuf/CBOR for smaller footprint and speed.
Example minimal REST publish (JSON):
POST /topics/telemetry/publish Content-Type: application/json Authorization: Bearer <token> {"device_id":"rpi-01","ts":1693440000,"temp":22.5}
Example WebSocket subscribe handshake (JSON):
{"action":"subscribe","topic":"telemetry","qos":"at-least-once"}
Delivery semantics and reliability
Designing predictable delivery guarantees in a tiny broker requires explicit choices:
- At-most-once: broker sends without expecting ack — low latency, possible message loss.
- At-least-once: broker requires ack and retries on timeout — duplicate deliveries possible.
- Best-effort: no guarantees, used for telemetry where occasional loss is acceptable.
Keep retries configurable (max attempts, backoff), and make message IDs available for de-duplication at consumers.
Storage format and compaction
If persistence is needed, an append-only log with simple segment files works well. Each segment contains length-prefixed messages with a small header (message id, timestamp, flags). Implement:
- Segment rotation by size or time.
- Periodic compaction for topics requiring retention by key (e.g., latest per device).
- Indexing: optional simple index file mapping message offsets to file positions for quick replay.
This approach mirrors Kafka’s log concept but trimmed for simplicity, without partitions or distributed coordination.
Security and access control (minimal)
- TLS for transport encryption.
- Simple token-based authentication (static API keys, short-lived JWT).
- Topic-level ACLs as a small optional layer: allow or deny publish/subscribe per token.
Avoid building a heavy RBAC system; integrate with existing identity providers if you need enterprise features.
Monitoring and observability
Essential metrics:
- Messages published/sent/acked per topic
- Consumer lag (if persisted)
- Heap and disk usage
- Errors and retry counts
Expose metrics via a Prometheus-compatible endpoint or a simple JSON /metrics endpoint. Keep logs structured and minimal.
Example implementation technologies
Languages and runtimes that fit RSMB goals:
- Go: small static binary, great concurrency primitives, strong ecosystem for networking.
- Rust: minimal runtime overhead, excellent for resource-constrained devices.
- C/C++: smallest possible footprint but higher implementation cost and maintenance.
- Node.js or Python: good for prototypes; larger memory profile but faster to iterate.
Choose lightweight libraries and avoid heavy frameworks. Statically link where possible for easier deployment.
Integration patterns
- Fan-out to local processors: devices publish telemetry; multiple local subscribers process or aggregate.
- Edge-to-cloud sync: local broker persists events and uploads to centralized broker or service when connectivity permits.
- Service mesh complement: use RSMB for intra-service events, while relying on centralized systems for long-term storage and analytics.
When not to use an RSMB
- You need multi-region replication, high throughput at scale, complex stream processing, or transactional semantics.
- Your team requires mature ecosystem features (connectors, schema registries, enterprise security).
- Regulatory requirements demand strict durability, auditing, and access controls.
In those cases, opt for a more feature-rich broker or hybrid architecture (use RSMB locally, central broker for long-term storage).
Example: Simple Go-based RSMB blueprint
High-level components:
- HTTP + WebSocket server
- In-memory topic registry with ring buffers
- Optional file-backed segments
- Ack tracker for at-least-once semantics
- Simple config for retention and QoS per topic
Pseudocode sketch (conceptual):
type Message struct { ID string TS int64 Body []byte } type Topic struct { name string queue *RingBuffer storage *SegmentStore // optional subs map[string]*Subscriber } func (t *Topic) Publish(msg Message) { if t.storage != nil { t.storage.Append(msg) } t.queue.Push(msg) for _, s := range t.subs { s.Push(msg) } }
Operational tips
- Start with in-memory mode for speed; enable persistence only where needed.
- Use small retention windows on edge devices to bound disk usage.
- Implement graceful shutdown that flushes in-memory messages to disk.
- Provide a compact admin CLI for operations (create/delete topic, inspect offsets).
Conclusion
A Really Small Message Broker is not an attempt to replace enterprise brokers but to fill a gap: a pragmatic, low-overhead messaging layer for small-scale, resource-constrained, or edge-first workloads. By focusing on minimal core features—simple pub/sub, optional durability, clear delivery semantics, and lightweight observability—you can deliver event streaming that’s fast to run, easy to operate, and cost-effective. Choose an RSMB when simplicity, resource efficiency, and predictable behavior are more valuable than advanced features and massive scale.