Apache Flink - Hardened Stream Processing Cluster

What is Apache Flink

Apache Flink is the open-source framework and distributed engine for stateful computations over unbounded and bounded data streams, from the Apache Software Foundation. It runs on the JVM and is deployed here as a single-node standalone session cluster: one JobManager that schedules and coordinates work and one TaskManager that executes it, on OpenJDK 17. Flink provides event-time processing, exactly-once state consistency, checkpoints and savepoints for fault tolerance, windowing and timers, and the layered DataStream, Table and SQL APIs with pluggable source and sink connectors. This image contains only the Apache-2.0 licensed distribution, with no proprietary add-ons and no vendor lock-in.

Why self-host Apache Flink

Running Flink on a VM you control keeps every job, its data and its computed state inside your own tenant rather than a managed streaming service. Self-hosting suits teams with data residency requirements, organisations under GDPR, HIPAA or ISO 27001, and any pipeline where the stream must stay within your perimeter with no per-record fees. Flink is Apache-2.0, fully auditable, with no vendor lock-in.

What this VM image adds

Security hardening:

• nginx TLS perimeter on port 443 - the Flink Web Dashboard has no native authentication, so it is reached only through nginx with HTTP Basic Auth; the admin password is generated uniquely at first boot
• Web Dashboard bound to 127.0.0.1:8081 and never exposed off-box - nginx is the only exposed door
• All internal cluster ports bound to localhost - JobManager RPC 6123, blob 6124, TaskManager 6121/6122 - not reachable from the network
• First-boot secrets only - the Basic Auth credentials are written to /root/flink-credentials.txt (mode 0600); nothing sensitive is baked into the image
• Self-signed TLS certificate generated at first launch and replaceable with your own CA-signed certificate (certbot is pre-installed)
• Non-root services - JobManager and TaskManager run as a dedicated flink system user under restricted systemd units; working and temporary directories are placed off the hardened noexec /tmp
• UFW firewall - TCP 443 and TCP 80 open externally for buyer use, TCP 22 for SSH; all other inbound dropped; Azure IMDS and WireServer egress pre-configured
• fail2ban - SSH brute-force protection
• AppArmor - mandatory access control
• CVE scan - every image is scanned with Trivy before release

OS hardening (CIS Level 1):

• CIS Ubuntu 24.04 LTS Level 1 Benchmark via ansible-lockdown
• auditd for system call auditing of critical paths
• SSH hardening - PasswordAuthentication disabled, key-only access, PermitRootLogin no, LoginGraceTime 60
• Kernel hardening - SYN cookies, ASLR, rp_filter, kexec disabled, IPv6 off
• /tmp as tmpfs with nosuid, nodev, noexec

Compliance artifacts (inside the VM):

• SBOM - CycloneDX 1.6 at /etc/lynxroute/sbom.json with Flink pinned by version, PURL, Apache-2.0 license, supplier, and hash
• CIS Conformance Report at /etc/lynxroute/cis-report.html (OpenSCAP, Azure tailoring profile, 0 FAIL rules)
• Tailored CIS profile at /usr/share/doc/lynxroute/CIS_TAILORED_PROFILE.md
• Operator credentials file at /root/flink-credentials.txt (mode 0600) with the Web Dashboard Basic Auth login and the service URL

Quick Start

1. Deploy VM from Azure Marketplace (Standard_D2s_v3 recommended)
2. Open NSG: TCP 443 (Web Dashboard) from your trusted sources, TCP 22 from your management IPs only
3. SSH: ssh -i key.pem azureuser@<PUBLIC_IP>, then sudo cat /root/flink-credentials.txt for the password
4. Open https://<PUBLIC_IP>/ in your browser, accept the self-signed certificate warning, and sign in with the Basic Auth user admin; the Apache Flink Dashboard loads
5. Submit a sample job: sudo -u flink /opt/flink/bin/flink run /opt/flink/examples/streaming/WordCount.jar, then watch it under Jobs > Completed in the Dashboard

The cluster is a single-node standalone session cluster (JobManager + TaskManager). The Web Dashboard is bound to 127.0.0.1:8081 and all internal cluster ports stay on localhost; nginx is the TLS perimeter on port 443 - do not expose 8081 directly. Replace the self-signed certificate with a CA-signed one. Checkpoints and savepoints are stored under /var/lib/flink.