From 16a774a3a57a4ff57fadca4dcc49e55b14b02158 Mon Sep 17 00:00:00 2001 From: Gyokhan Kochmarla Date: Mon, 23 Feb 2026 09:52:54 +0100 Subject: [PATCH 1/3] docs(readme): overhaul README with visual assets, Docker, and storytelling Rewrite the README to tell a compelling story about why ProtoMQ exists and what makes it useful for IoT/edge deployments: - New tagline emphasizing MQTT + Protobuf + Zig synergy - Added SVG payload comparison chart (JSON vs Protobuf, 3 scenarios) - Added SVG terminal demo showing the publish/subscribe workflow - Mermaid architecture diagram of the broker internals - Reframed performance benchmarks with context - Docker quick-start with Dockerfile and docker-compose.yml - Condensed Service Discovery and Admin Server sections - Navigation links and FAQ reference - Added .blog/ to .gitignore for draft social media posts Signed-off-by: Gyokhan Kochmarla --- .gitignore | 3 + Dockerfile | 32 +++++ README.md | 224 +++++++++++++++++++++++++--------- assets/payload_comparison.svg | 71 +++++++++++ assets/terminal_demo.svg | 77 ++++++++++++ docker-compose.yml | 8 ++ 6 files changed, 360 insertions(+), 55 deletions(-) create mode 100644 Dockerfile create mode 100644 assets/payload_comparison.svg create mode 100644 assets/terminal_demo.svg create mode 100644 docker-compose.yml diff --git a/.gitignore b/.gitignore index 74233b8..44207f7 100644 --- a/.gitignore +++ b/.gitignore @@ -36,3 +36,6 @@ coverage/ # Dependency cache deps/ vendor/ + +# Blog drafts (not part of the repo) +.blog/ diff --git a/Dockerfile b/Dockerfile new file mode 100644 index 0000000..6444ee4 --- /dev/null +++ b/Dockerfile @@ -0,0 +1,32 @@ +FROM alpine:3.21 AS builder + +# Install Zig 0.15.2 and build dependencies. +RUN apk add --no-cache curl xz && \ + curl -L https://ziglang.org/download/0.15.2/zig-x86_64-linux-0.15.2.tar.xz | \ + tar -xJ -C /opt && \ + mv /opt/zig-x86_64-linux-0.15.2 /opt/zig +ENV PATH="/opt/zig:${PATH}" + +WORKDIR /src +COPY build.zig build.zig.zon ./ +COPY src/ src/ +COPY schemas/ schemas/ + +RUN zig build -Doptimize=ReleaseSafe -Dtarget=x86_64-linux + +# --- Runtime stage --- +FROM alpine:3.21 + +RUN addgroup -S protomq && adduser -S protomq -G protomq + +WORKDIR /opt/protomq + +COPY --from=builder /src/zig-out/bin/protomq-server ./bin/protomq-server +COPY --from=builder /src/zig-out/bin/protomq-cli ./bin/protomq-cli +COPY schemas/ ./schemas/ + +USER protomq + +EXPOSE 1883 + +ENTRYPOINT ["./bin/protomq-server"] diff --git a/README.md b/README.md index f1fad78..da2c89b 100644 --- a/README.md +++ b/README.md @@ -2,100 +2,214 @@

ProtoMQ

- ProtoMQ Mascot + ProtoMQ Mascot
- Type-safe, bandwidth-efficient MQTT for the rest of us. + MQTT's simplicity. Protobuf's efficiency. Zig's bare-metal performance.
- Stop sending bloated JSON over the wire. + Built for IoT and edge computing.

+

+ Quick Start • + Why ProtoMQ • + Architecture • + Performance • + FAQ +

+ +--- + +## What It Does + +ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. Clients publish JSON; the broker validates, encodes to Protobuf, and routes compact binary payloads to subscribers. No code generation, no `.proto` compilation step on the client side. + +

+ ProtoMQ terminal demo +

+ +- **Schema-enforced messaging** — `.proto` files define the contract. Malformed payloads are rejected *before* they reach subscribers. +- **Custom Protobuf engine** — written from scratch in Zig. Runtime `.proto` parsing with zero external dependencies. +- **Wildcard topic routing** — full MQTT `+` and `#` wildcard support with O(1) matching via a trie-based topic broker. +- **Service Discovery** — clients query `$SYS/discovery/request` to discover available topics and download schemas automatically. No pre-shared `.proto` files needed. +- **Runs in 2.6 MB** — the entire broker with 100 active connections fits in under 3 MB of memory. + +--- + +## Why ProtoMQ + +Every IoT team eventually faces the same problem: JSON payloads are readable but *wasteful*. A 12-field sensor reading that takes 310 bytes in JSON compresses to 82 bytes in Protobuf — that's a **74% reduction** in bandwidth, which directly translates to lower radio usage, longer battery life, and cheaper data plans for cellular-connected devices. + +

+ JSON vs Protobuf payload size comparison +

+ +But switching to Protobuf introduces its own headache: you need code generation, language-specific stubs, and a way to keep producers and consumers in sync when schemas change. ProtoMQ removes that friction: + +| Concern | Plain MQTT + JSON | ProtoMQ | +|---|---|---| +| Schema enforcement | None — anything goes | Validated at every `PUBLISH` | +| Payload format | JSON (~170 bytes for 8 fields) | Protobuf (~48 bytes) | +| Client bootstrap | Pre-shared docs, out-of-band | Built-in Service Discovery | +| Code generation | Required per language | Not needed — send JSON, receive Protobuf | +| Observability | Roll your own | Admin HTTP API included | + +--- + +## Architecture + +```mermaid +graph LR + subgraph Clients + P[Producer
sends JSON] + S1[Subscriber 1] + S2[Subscriber 2] + end + + subgraph ProtoMQ Broker + TCP[TCP Listener
epoll / kqueue] + MP[MQTT Parser
v3.1.1] + PB[Protobuf Engine
validate + encode] + TB[Topic Broker
wildcard trie] + SR[Schema Registry
.proto files] + end + + P -->|MQTT CONNECT + PUBLISH| TCP + TCP --> MP + MP --> PB + PB -->|lookup| SR + PB -->|binary payload| TB + TB --> S1 + TB --> S2 + + style PB fill:#58a6ff,color:#0d1117 + style SR fill:#3fb950,color:#0d1117 + style TB fill:#f97316,color:#0d1117 +``` + +The broker is a single-threaded event loop built on `epoll` (Linux) or `kqueue` (macOS). There are no hidden allocations, no garbage collector, and no runtime — just Zig's `std.mem.Allocator` with explicit control over every byte. + +The Protobuf engine parses `.proto` files at startup (or at runtime via the Admin API) and builds an in-memory schema registry. When a PUBLISH arrives, the engine validates the JSON payload against the registered schema, encodes it to Protobuf wire format, and hands the compact binary to the topic broker for fan-out. + --- -- MQTT v3.1.1 packet parsing (CONNECT, PUBLISH, SUBSCRIBE, etc.) -- Thread-safe Topic Broker with wildcard support (`+`, `#`) -- Custom Protobuf Engine with runtime `.proto` schema parsing -- Topic-based Protobuf schema routing -- Service Discovery & Schema Registry -- CLI with automatic JSON-to-Protobuf encoding -- Structured diagnostic output for Protobuf payloads -### Building +## Quick Start -One have to have Zig 0.15.2 or later installed. Please download it from [here](https://ziglang.org/download/). +### Docker (recommended for trying it out) ```bash -# Build server and client -zig build +docker compose up +``` + +The server starts on port `1883` with the schemas from the `schemas/` directory. Connect with any MQTT client. + +### Build from source + +Requires [Zig 0.15.2+](https://ziglang.org/download/). -# Build and run server +```bash +git clone https://github.com/electricalgorithm/protomq.git +cd protomq + +# Build and run the server zig build run-server -# Build and run client -zig build run-client +# In another terminal — publish a JSON message +zig build run-client -- publish --topic sensors/temp \ + --json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}' + +# In another terminal — subscribe to all sensor topics +zig build run-client -- subscribe --topic "sensors/#" +``` + +### Run tests -# Run tests +```bash +# Unit tests zig build test -# Run all integration tests +# Full integration suite ./tests/run_all.sh ``` -### Limitations +--- -For the initial release, we support: -- QoS 0 only (at most once delivery) -- No persistent sessions -- No retained messages -- Single-node deployment +## Performance -### Service Discovery +ProtoMQ handles **208,000 messages/second** on an Apple M2 Pro and **147,000 msg/s** on a Raspberry Pi 5 — with sub-millisecond p99 latency and no memory leaks across 100,000 connection cycles. -ProtoMQ includes a built-in Service Discovery mechanism. Clients can discover available topics and their associated Protobuf schemas (including the full source code) by querying the `$SYS/discovery/request` topic. +| Scenario | Apple M2 Pro | Raspberry Pi 5 | +|----------|--------------|----------------| +| **p99 latency** (100 clients) | 0.44 ms | 0.13 ms | +| **Throughput** (10-byte msgs) | 208k msg/s | 147k msg/s | +| **Throughput** (64 KB msgs) | 39k msg/s | 27k msg/s | +| **Sustained load** (10 min) | 8,981 msg/s | 9,012 msg/s | +| **Memory** (100 connections) | 2.6 MB | 2.5 MB | +| **Connection churn** (100k cycles) | 1,496 conn/s | 1,548 conn/s | +| **Memory leaks** | 0 MB | 0 MB | + +All benchmarks run on loopback, `ReleaseSafe` mode, Zig 0.15.2. Methodology and raw JSON results: [`benchmarks/`](benchmarks/README.md). + +--- + +## Service Discovery + +Clients can discover topics and download the associated `.proto` schemas without any out-of-band configuration: -**Using the CLI for discovery:** ```bash -# Verify schemas are loaded and available protomq-cli discover --proto-dir schemas ``` -This allows clients to "bootstrap" themselves without needing pre-shared `.proto` files. -### Admin Server +Under the hood, the client subscribes to `$SYS/discovery/request` and receives a `ServiceDiscoveryResponse` containing every registered topic-schema mapping including the full `.proto` source code. This lets new clients bootstrap themselves in a single round-trip. + +--- + +## Admin Server + +An optional HTTP server for runtime schema management and telemetry. Disabled by default — when the build flag is off, the code is stripped from the binary entirely (zero overhead, not just disabled). + +```bash +# Enable at compile time +zig build -Dadmin_server=true run-server +``` + +| Endpoint | Description | +|---|---| +| `GET /metrics` | Active connections, message throughput, loaded schemas | +| `GET /api/v1/schemas` | Current topic-to-schema mappings | +| `POST /api/v1/schemas` | Register a new `.proto` schema and topic mapping at runtime | -ProtoMQ includes an optional HTTP Admin Server for runtime observability and dynamic schema management without polluting the core MQTT hot-paths. +All endpoints require `Authorization: Bearer `. The Admin Server runs cooperatively on the same event loop — enabling it does not degrade MQTT performance. -- **Dynamic Schema Registration**: Register `.proto` files at runtime via `POST /api/v1/schemas`. -- **Telemetry**: Monitor active connections, message throughput, and schemas via `GET /metrics`. -- **Zero Overhead Footprint**: The Admin Server is disabled by default to preserve the absolute minimum memory footprint for embedded devices. It is strictly conditionally compiled via the `zig build -Dadmin_server=true` flag. Enabling it moderately increases the initial static memory baseline (e.g., from ~2.6 MB to ~4.0 MB) by safely running a parallel HTTP listener, but it executes cooperatively on the same event loop ensuring zero degradation to per-message MQTT performance. When the flag is deactivated, it incurs **zero overhead footprint**. +| Build | Memory baseline | Admin API | +|---|---|---| +| `zig build` | ~2.6 MB | ✗ | +| `zig build -Dadmin_server=true` | ~4.0 MB | ✓ | -### Performance Results +For detailed usage including dynamic schema registration: [FAQ](FAQ.md). -ProtoMQ delivers high performance across both high-end and edge hardware: +--- -| Scenario | Apple M2 Pro | Raspberry Pi 5 | -|----------|--------------|----------------| -| Latency (p99, 100 clients) | 0.44 ms | 0.13 ms | -| Concurrent clients | 10,000 | 10,000 | -| Sustained throughput | 9k msg/s | 9k msg/s | -| Message throughput (small) | 208k msg/s | 147k msg/s | -| Memory (100 clients) | 2.6 MB | 2.5 MB | +## Current Limitations -Handles 100,000 connection cycles with zero memory leaks and sub-millisecond latency. +- QoS 0 only (at most once delivery) +- No persistent sessions +- No retained messages +- Single-node deployment + +These are scope decisions for the initial release, not fundamental limitations. Multi-node and QoS 1/2 are on the roadmap. -For detailed methodology and full results, see [ProtoMQ Benchmarking Suite](benchmarks/README.md). +--- -### Contributing +## Contributing -This is currently a learning/development project. Contributions will be welcome after the MVP is complete. +Contributions are welcome! This project is under active development — if you're interested in MQTT internals, Protobuf wire format, or systems programming in Zig, there's plenty to work on. -### License +## License -The project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. +MIT — see [LICENSE](LICENSE). -### Resources +## Resources - [Zig Documentation](https://ziglang.org/documentation/master/) - [MQTT v3.1.1 Specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html) - [Protocol Buffers](https://protobuf.dev/) - ---- - -**Note**: This project is under active development. The API and architecture may change significantly. diff --git a/assets/payload_comparison.svg b/assets/payload_comparison.svg new file mode 100644 index 0000000..5490b6f --- /dev/null +++ b/assets/payload_comparison.svg @@ -0,0 +1,71 @@ + + + + + + + + + Payload Size: JSON vs Protobuf + Identical sensor telemetry data — fewer bytes = less bandwidth, lower latency + + + SensorData + 4 fields · device_id, temp, humidity, ts + + + JSON + + 114 bytes + + + Protobuf + + 40 bytes + ▼ 65% smaller + + + SensorTelemetry + 8 fields · device_id, temp, humidity, pressure, battery, lat, lon, ts + + + JSON + + 170 bytes + + + Protobuf + + 48 bytes + ▼ 72% smaller + + + FleetStatus + 12 fields · id, coords, engine metrics, diagnostics, timestamp + + + JSON + + 310 bytes + + + Protobuf + + 82 bytes + ▼ 74% smaller + + + + JSON (text) + + Protobuf (binary) + diff --git a/assets/terminal_demo.svg b/assets/terminal_demo.svg new file mode 100644 index 0000000..440f278 --- /dev/null +++ b/assets/terminal_demo.svg @@ -0,0 +1,77 @@ + + + + + + + + + + + + + + protomq — zsh — 120×30 + + + ❯ + zig build run-server -- --proto-dir schemas + + + [info] + ProtoMQ v0.1.0 starting on 0.0.0.0:1883 + [info] + Loaded schema: iot.sensor.SensorData (4 fields, 40 bytes max) + [info] + Mapped topic "sensors/#" → SensorData + [info] + Event loop ready (kqueue backend, 1000 max connections) + + + ─────────────────────────────────── terminal 2 ────────────────────────────────── + + + # Publish a JSON payload — ProtoMQ encodes it to Protobuf automatically + ❯ + protomq-cli publish --topic sensors/temp \ + --json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}' + + + [protobuf] + Encoded 114-byte JSON → 28-byte Protobuf (75% reduction) + [publish] + sensors/temp → 1 subscriber(s), 28 bytes delivered + + + ─────────────────────────────────── terminal 3 ────────────────────────────────── + + + # Subscribe and see decoded messages in real time + ❯ + protomq-cli subscribe --topic "sensors/#" + + + [sensors/temp] + SensorData { + device_id: "s-042" + temperature: 22.5 + humidity: 61 + timestamp: 1706140800 + } + diff --git a/docker-compose.yml b/docker-compose.yml new file mode 100644 index 0000000..0c3cc09 --- /dev/null +++ b/docker-compose.yml @@ -0,0 +1,8 @@ +services: + protomq: + build: . + ports: + - "1883:1883" + volumes: + - ./schemas:/opt/protomq/schemas:ro + restart: unless-stopped From e840500ff48e350f3dd1681908b80fc1c3be124f Mon Sep 17 00:00:00 2001 From: Gyokhan Kochmarla Date: Mon, 23 Feb 2026 10:10:37 +0100 Subject: [PATCH 2/3] docs(readme): simplify structure, add FEATURES.md for detailed docs - Replace Mermaid architecture diagram with 'Under the Hood' section highlighting epoll/kqueue, zero deps, runtime schema registry, single allocator, comptime tables, and cross-compilation - Use h3 headers instead of h2 to reduce visual noise - Remove Resources, License, and standalone Service Discovery sections - Condense Admin Server to a one-line mention with link to FEATURES.md - Rewrite 'Why ProtoMQ' in a more conversational tone - Move detailed Service Discovery and Admin Server docs to FEATURES.md Signed-off-by: Gyokhan Kochmarla --- FEATURES.md | 69 ++++++++++++++++++++++ README.md | 163 ++++++++++++---------------------------------------- 2 files changed, 105 insertions(+), 127 deletions(-) create mode 100644 FEATURES.md diff --git a/FEATURES.md b/FEATURES.md new file mode 100644 index 0000000..7d7f2c5 --- /dev/null +++ b/FEATURES.md @@ -0,0 +1,69 @@ +# Features + +This document covers ProtoMQ's features in more depth than the README. If you're looking for deployment guides and configuration, check [FAQ.md](FAQ.md). + +--- + +## Service Discovery + +ProtoMQ includes a built-in Service Discovery mechanism. Clients can discover available topics and their associated Protobuf schemas — including the full `.proto` source code — by querying the `$SYS/discovery/request` topic. + +This lets new clients bootstrap themselves in a single round-trip without any out-of-band configuration or pre-shared schema files. + +**Using the CLI:** + +```bash +protomq-cli discover --proto-dir schemas +``` + +Under the hood, the client subscribes to `$SYS/discovery/request` and receives a `ServiceDiscoveryResponse` message containing every registered topic-schema mapping. The response includes the raw `.proto` source so clients can dynamically configure their own decoding logic. + +--- + +## Admin Server + +An optional HTTP server for runtime schema management and telemetry. Disabled by default — when the build flag is off, the HTTP code is **completely stripped from the binary** (zero overhead, not just disabled). + +### Enabling it + +```bash +zig build -Dadmin_server=true run-server +``` + +### Build comparison + +| Build | Memory baseline | Admin API | +|---|---|---| +| `zig build` | ~2.6 MB | ✗ | +| `zig build -Dadmin_server=true` | ~4.0 MB | ✓ | + +The Admin Server runs cooperatively on the same event loop as the MQTT broker — enabling it does **not** degrade per-message MQTT performance. + +### Endpoints + +All endpoints require `Authorization: Bearer ` (defaults to `admin_secret`, configurable via the `ADMIN_TOKEN` environment variable). + +| Method | Path | Description | +|---|---|---| +| `GET` | `/metrics` | Active connections, message throughput, loaded schemas | +| `GET` | `/api/v1/schemas` | Current topic-to-schema mappings | +| `POST` | `/api/v1/schemas` | Register a new `.proto` schema and map it to a topic at runtime | + +### Dynamic schema registration + +With the Admin Server enabled, you can register schemas at runtime without restarting the broker: + +```bash +curl -X POST http://127.0.0.1:8080/api/v1/schemas \ + -H "Authorization: Bearer ${ADMIN_TOKEN:-admin_secret}" \ + -H "Content-Type: application/json" \ + -d '{ + "topic": "telemetry/gps", + "message_type": "GpsCoordinate", + "proto_file_content": "syntax = \"proto3\";\nmessage GpsCoordinate { float lat = 1; float lon = 2; }" + }' +``` + +The schema is parsed in-process and persisted to disk as `schemas/.proto`. The mapping is live immediately — no restart needed. + +> **Security note**: The Admin Server binds to `127.0.0.1:8080` only. If you need remote access, use an SSH tunnel or reverse proxy. diff --git a/README.md b/README.md index da2c89b..89ee26a 100644 --- a/README.md +++ b/README.md @@ -12,128 +12,92 @@

Quick Start • Why ProtoMQ • - Architecture • Performance • + Features • FAQ

--- -## What It Does - ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. Clients publish JSON; the broker validates, encodes to Protobuf, and routes compact binary payloads to subscribers. No code generation, no `.proto` compilation step on the client side.

ProtoMQ terminal demo

-- **Schema-enforced messaging** — `.proto` files define the contract. Malformed payloads are rejected *before* they reach subscribers. -- **Custom Protobuf engine** — written from scratch in Zig. Runtime `.proto` parsing with zero external dependencies. -- **Wildcard topic routing** — full MQTT `+` and `#` wildcard support with O(1) matching via a trie-based topic broker. -- **Service Discovery** — clients query `$SYS/discovery/request` to discover available topics and download schemas automatically. No pre-shared `.proto` files needed. +- **Schema-enforced messaging** — `.proto` files define the contract. Malformed payloads get rejected *before* they reach subscribers. +- **Custom Protobuf engine** — written from scratch in Zig. Runtime `.proto` parsing, zero external dependencies. +- **Wildcard topic routing** — full MQTT `+` and `#` wildcard support via a trie-based topic broker. +- **Service Discovery** — clients query `$SYS/discovery/request` to discover topics and download schemas on the fly. No pre-shared `.proto` files needed. +- **Optional Admin HTTP API** — register new schemas and topic mappings at runtime, monitor connections and throughput. Disabled by default, zero overhead when off. See [FEATURES.md](FEATURES.md) for details. - **Runs in 2.6 MB** — the entire broker with 100 active connections fits in under 3 MB of memory. --- -## Why ProtoMQ +### Why ProtoMQ + +If you've worked with IoT sensor fleets, you've probably been through this: you start with JSON over MQTT because it's easy to debug, every language has a parser, and `mosquitto_sub` lets you eyeball what's going on. It works fine... until you start caring about bandwidth. -Every IoT team eventually faces the same problem: JSON payloads are readable but *wasteful*. A 12-field sensor reading that takes 310 bytes in JSON compresses to 82 bytes in Protobuf — that's a **74% reduction** in bandwidth, which directly translates to lower radio usage, longer battery life, and cheaper data plans for cellular-connected devices. +A 12-field sensor reading weighs around 310 bytes in JSON. The same data in Protobuf: 82 bytes. On a cellular-connected device pushing telemetry every 5 seconds, that gap adds up to roughly 1.6 MB/day per device — multiply by a few thousand devices and the data bill starts hurting.

JSON vs Protobuf payload size comparison

-But switching to Protobuf introduces its own headache: you need code generation, language-specific stubs, and a way to keep producers and consumers in sync when schemas change. ProtoMQ removes that friction: +But switching to Protobuf means you need code generation per language, keeping stubs in sync across firmware versions, and losing the ability to just read your payloads. ProtoMQ sits in the middle: producers send plain JSON, the broker handles encoding, and subscribers get compact Protobuf. The schema lives in one place and the broker enforces it. -| Concern | Plain MQTT + JSON | ProtoMQ | +| | Plain MQTT + JSON | ProtoMQ | |---|---|---| | Schema enforcement | None — anything goes | Validated at every `PUBLISH` | -| Payload format | JSON (~170 bytes for 8 fields) | Protobuf (~48 bytes) | -| Client bootstrap | Pre-shared docs, out-of-band | Built-in Service Discovery | -| Code generation | Required per language | Not needed — send JSON, receive Protobuf | -| Observability | Roll your own | Admin HTTP API included | +| Payload format | JSON (~170 bytes, 8 fields) | Protobuf (~48 bytes) | +| Client bootstrap | Pre-shared docs | Built-in Service Discovery | +| Code generation | Required per language | Not needed | --- -## Architecture - -```mermaid -graph LR - subgraph Clients - P[Producer
sends JSON] - S1[Subscriber 1] - S2[Subscriber 2] - end - - subgraph ProtoMQ Broker - TCP[TCP Listener
epoll / kqueue] - MP[MQTT Parser
v3.1.1] - PB[Protobuf Engine
validate + encode] - TB[Topic Broker
wildcard trie] - SR[Schema Registry
.proto files] - end - - P -->|MQTT CONNECT + PUBLISH| TCP - TCP --> MP - MP --> PB - PB -->|lookup| SR - PB -->|binary payload| TB - TB --> S1 - TB --> S2 - - style PB fill:#58a6ff,color:#0d1117 - style SR fill:#3fb950,color:#0d1117 - style TB fill:#f97316,color:#0d1117 -``` +### Under the Hood -The broker is a single-threaded event loop built on `epoll` (Linux) or `kqueue` (macOS). There are no hidden allocations, no garbage collector, and no runtime — just Zig's `std.mem.Allocator` with explicit control over every byte. +ProtoMQ is not a wrapper around an existing broker — it's a ground-up implementation. Here's what makes it tick: -The Protobuf engine parses `.proto` files at startup (or at runtime via the Admin API) and builds an in-memory schema registry. When a PUBLISH arrives, the engine validates the JSON payload against the registered schema, encodes it to Protobuf wire format, and hands the compact binary to the topic broker for fan-out. +- **`epoll` / `kqueue` event loop** — single-threaded, no abstraction layer. The network layer talks directly to the OS kernel I/O primitives. On Linux that's `epoll`, on macOS `kqueue`. No libuv, no tokio, no hidden threads. +- **One allocator, full control** — every allocation goes through Zig's `std.mem.Allocator`. No GC, no hidden heap churn, no runtime. You can trace every byte the broker touches. +- **Zero third-party dependencies** — the MQTT parser, TCP connection handler, Protobuf wire format encoder, `.proto` file parser — all written in Zig using only the standard library. `build.zig.zon` has an empty `dependencies` block. +- **Runtime schema registry** — `.proto` files are parsed at startup and mapped to MQTT topics. With the Admin Server enabled, you can register new schemas and mappings at runtime over HTTP without restarting the broker. +- **Comptime-generated lookup tables** — the MQTT packet parser uses Zig's `comptime` to build dispatch tables at compile time. No branching, no hash maps — just array indexing. +- **Cross-compilation** — `zig build -Dtarget=aarch64-linux` produces a Raspberry Pi binary from a Mac. One command, no toolchain headaches. --- -## Quick Start +### Quick Start -### Docker (recommended for trying it out) +**Docker:** ```bash docker compose up ``` -The server starts on port `1883` with the schemas from the `schemas/` directory. Connect with any MQTT client. +The server starts on port `1883` with the schemas from `schemas/`. Connect with any MQTT client. -### Build from source - -Requires [Zig 0.15.2+](https://ziglang.org/download/). +**From source** (requires [Zig 0.15.2+](https://ziglang.org/download/)): ```bash git clone https://github.com/electricalgorithm/protomq.git cd protomq - -# Build and run the server zig build run-server +``` +```bash # In another terminal — publish a JSON message zig build run-client -- publish --topic sensors/temp \ --json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}' -# In another terminal — subscribe to all sensor topics +# In another terminal — subscribe zig build run-client -- subscribe --topic "sensors/#" ``` -### Run tests - -```bash -# Unit tests -zig build test - -# Full integration suite -./tests/run_all.sh -``` - --- -## Performance +### Performance ProtoMQ handles **208,000 messages/second** on an Apple M2 Pro and **147,000 msg/s** on a Raspberry Pi 5 — with sub-millisecond p99 latency and no memory leaks across 100,000 connection cycles. @@ -147,69 +111,14 @@ ProtoMQ handles **208,000 messages/second** on an Apple M2 Pro and **147,000 msg | **Connection churn** (100k cycles) | 1,496 conn/s | 1,548 conn/s | | **Memory leaks** | 0 MB | 0 MB | -All benchmarks run on loopback, `ReleaseSafe` mode, Zig 0.15.2. Methodology and raw JSON results: [`benchmarks/`](benchmarks/README.md). - ---- - -## Service Discovery - -Clients can discover topics and download the associated `.proto` schemas without any out-of-band configuration: - -```bash -protomq-cli discover --proto-dir schemas -``` - -Under the hood, the client subscribes to `$SYS/discovery/request` and receives a `ServiceDiscoveryResponse` containing every registered topic-schema mapping including the full `.proto` source code. This lets new clients bootstrap themselves in a single round-trip. +All benchmarks run on loopback, `ReleaseSafe` mode, Zig 0.15.2. Methodology and raw results: [`benchmarks/`](benchmarks/README.md). --- -## Admin Server - -An optional HTTP server for runtime schema management and telemetry. Disabled by default — when the build flag is off, the code is stripped from the binary entirely (zero overhead, not just disabled). - -```bash -# Enable at compile time -zig build -Dadmin_server=true run-server -``` - -| Endpoint | Description | -|---|---| -| `GET /metrics` | Active connections, message throughput, loaded schemas | -| `GET /api/v1/schemas` | Current topic-to-schema mappings | -| `POST /api/v1/schemas` | Register a new `.proto` schema and topic mapping at runtime | - -All endpoints require `Authorization: Bearer `. The Admin Server runs cooperatively on the same event loop — enabling it does not degrade MQTT performance. - -| Build | Memory baseline | Admin API | -|---|---|---| -| `zig build` | ~2.6 MB | ✗ | -| `zig build -Dadmin_server=true` | ~4.0 MB | ✓ | - -For detailed usage including dynamic schema registration: [FAQ](FAQ.md). - ---- - -## Current Limitations - -- QoS 0 only (at most once delivery) -- No persistent sessions -- No retained messages -- Single-node deployment - -These are scope decisions for the initial release, not fundamental limitations. Multi-node and QoS 1/2 are on the roadmap. - ---- - -## Contributing - -Contributions are welcome! This project is under active development — if you're interested in MQTT internals, Protobuf wire format, or systems programming in Zig, there's plenty to work on. - -## License +### Current Limitations -MIT — see [LICENSE](LICENSE). +QoS 0 only (at most once delivery), no persistent sessions, no retained messages, single-node deployment. These are scope decisions for the initial release — multi-node and QoS 1/2 are on the roadmap. -## Resources +### Contributing -- [Zig Documentation](https://ziglang.org/documentation/master/) -- [MQTT v3.1.1 Specification](http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/mqtt-v3.1.1.html) -- [Protocol Buffers](https://protobuf.dev/) +Contributions are welcome. If you're interested in MQTT internals, Protobuf wire format, or systems programming in Zig, there's plenty to dig into. See [FEATURES.md](FEATURES.md) for the full feature set and [FAQ.md](FAQ.md) for deployment and configuration guides. From 65cd9395045c84f20562319ba4510d6e237d8f0d Mon Sep 17 00:00:00 2001 From: Gyokhan Kochmarla Date: Tue, 24 Feb 2026 08:44:53 +0100 Subject: [PATCH 3/3] =?UTF-8?q?fix(readme):=20correct=20protocol=20descrip?= =?UTF-8?q?tion=20=E2=80=94=20Protobuf-native,=20not=20JSON-to-Protobuf?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The broker only accepts Protobuf payloads on the wire. The CLI has a --json convenience flag that encodes to Protobuf client-side before publishing, but the protocol itself is Protobuf-only. The previous wording incorrectly implied the broker does JSON-to-Protobuf conversion. Signed-off-by: Gyokhan Kochmarla --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 89ee26a..96b485e 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ --- -ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. Clients publish JSON; the broker validates, encodes to Protobuf, and routes compact binary payloads to subscribers. No code generation, no `.proto` compilation step on the client side. +ProtoMQ is an MQTT broker that enforces **Protobuf schemas at the broker level**. All messages on the wire are Protobuf — the broker validates incoming payloads against registered `.proto` schemas and rejects anything that doesn't conform. The bundled CLI can accept JSON and encode it to Protobuf client-side for convenience.

ProtoMQ terminal demo @@ -44,14 +44,14 @@ A 12-field sensor reading weighs around 310 bytes in JSON. The same data in Prot JSON vs Protobuf payload size comparison

-But switching to Protobuf means you need code generation per language, keeping stubs in sync across firmware versions, and losing the ability to just read your payloads. ProtoMQ sits in the middle: producers send plain JSON, the broker handles encoding, and subscribers get compact Protobuf. The schema lives in one place and the broker enforces it. +But switching to Protobuf usually means code generation per language, keeping stubs in sync across firmware versions, and losing the ability to just read your payloads. ProtoMQ takes a different approach: the broker owns the `.proto` schemas and validates every message against them. The CLI can accept JSON and encode it to Protobuf before publishing, so you get a human-friendly workflow without sacrificing wire efficiency. | | Plain MQTT + JSON | ProtoMQ | |---|---|---| | Schema enforcement | None — anything goes | Validated at every `PUBLISH` | | Payload format | JSON (~170 bytes, 8 fields) | Protobuf (~48 bytes) | | Client bootstrap | Pre-shared docs | Built-in Service Discovery | -| Code generation | Required per language | Not needed | +| Code generation | Required per language | CLI encodes JSON → Protobuf for you | --- @@ -87,7 +87,7 @@ zig build run-server ``` ```bash -# In another terminal — publish a JSON message +# In another terminal — publish (CLI encodes JSON to Protobuf for you) zig build run-client -- publish --topic sensors/temp \ --json '{"device_id":"s-042","temperature":22.5,"humidity":61,"timestamp":1706140800}'