Apache Kafka - Hardened Event Streaming with Kafbat UI

What is Apache Kafka

Apache Kafka is the widely deployed distributed event streaming platform from the Apache Software Foundation. It powers high-throughput, low-latency data pipelines, messaging, event sourcing and stream processing over an append-only commit log, with durable topic storage, consumer groups and partition-level ordering. Kafka 4.x runs entirely in KRaft mode (the Kafka Raft metadata protocol) - ZooKeeper has been removed - so this image ships a single, self-contained combined controller+broker node on OpenJDK 21. It contains only the Apache-2.0 licensed distribution, with no proprietary add-ons and no vendor lock-in.

Why self-host Apache Kafka

Running Kafka on a VM you control keeps every event - often sensitive customer, transactional or operational data - inside your own tenant rather than a managed streaming service. Self-hosting suits teams with data residency requirements, organisations under GDPR, HIPAA or ISO 27001, and any product where the event stream must stay within your perimeter with no per-message fees. Kafka is Apache-2.0, fully auditable, with no vendor lock-in.

What this VM image adds

Management UI:

• Kafbat UI (Apache-2.0) bundled as a systemd service - browse topics, consumer groups, messages and broker configuration from your browser
• nginx TLS perimeter - the UI is bound to loopback and reached only through nginx on port 443 with login-form authentication; the admin password is generated at first boot

Security hardening:

• Internal broker listener bound to 127.0.0.1:9092 (PLAINTEXT, used only by the bundled Kafbat UI) - never exposed off-box
• External SASL_SSL listener on port 9093 - clients authenticate with SCRAM-SHA-256 over TLS; the client user, password and self-signed broker keystore are generated uniquely at first boot
• First-boot secrets only - credentials written to /root/kafka-credentials.txt and /root/kafka-client.properties (mode 0600); nothing sensitive is baked into the image
• Self-signed UI certificate generated at first launch and replaceable with your own CA-signed certificate (certbot is pre-installed)
• Non-root service - the broker runs as a dedicated kafka system user with UMask 0027; JVM heaps set explicitly
• UFW firewall - TCP 443 and TCP 9093 open externally for buyer use, TCP 22 for SSH; all other inbound dropped; Azure IMDS and WireServer egress pre-configured
• fail2ban - SSH brute-force protection
• AppArmor - mandatory access control
• CVE scan - every image is scanned with Trivy before release

OS hardening (CIS Level 1):

• CIS Ubuntu 24.04 LTS Level 1 Benchmark via ansible-lockdown
• auditd for system call auditing of critical paths
• SSH hardening - PasswordAuthentication disabled, key-only access, PermitRootLogin no, LoginGraceTime 60
• Kernel hardening - SYN cookies, ASLR, rp_filter, kexec disabled, IPv6 off
• /tmp as tmpfs with nosuid, nodev, noexec

Compliance artifacts (inside the VM):

• SBOM - CycloneDX 1.6 at /etc/lynxroute/sbom.json with Kafka pinned by version, PURL, Apache-2.0 license, supplier, and hash
• CIS Conformance Report at /etc/lynxroute/cis-report.html (OpenSCAP, Azure tailoring profile, 0 FAIL rules)
• Tailored CIS profile at /usr/share/doc/lynxroute/CIS_TAILORED_PROFILE.md
• Operator credentials file at /root/kafka-credentials.txt (mode 0600) with the Kafbat UI admin login, the SCRAM client user, and the service URLs

Quick Start

1. Deploy VM from Azure Marketplace (Standard_D2s_v3 recommended)
2. Open NSG: TCP 443 (Kafbat UI) and TCP 9093 (Kafka clients) from your trusted sources, TCP 22 from your management IPs only
3. SSH: ssh -i key.pem azureuser@<PUBLIC_IP>, then sudo cat /root/kafka-credentials.txt for the passwords
4. Open https://<PUBLIC_IP>/ in your browser, accept the self-signed certificate warning, and log in to Kafbat UI with the admin user
5. Connect an external client over the SASL_SSL listener on port 9093 using the generated SCRAM user; the client config is written to /root/kafka-client.properties

The broker runs single-node KRaft (no ZooKeeper). The internal PLAINTEXT listener is bound to 127.0.0.1:9092 and used only by the bundled Kafbat UI; nginx is the TLS perimeter on port 443 - do not expose 8080 or 9092 directly. Replace the self-signed certificate with a CA-signed one.