Apache Airflow - Hardened Workflow Orchestration

What is Apache Airflow

Apache Airflow is the leading open-source platform to programmatically author, schedule, and monitor workflows - the de facto standard for orchestrating data pipelines. Workflows are defined as Directed Acyclic Graphs (DAGs) in plain Python, giving you dynamic pipeline generation, flexible scheduling, backfills, retries, branching, and a web UI for monitoring task runs and logs. Airflow ships hundreds of provider packages that integrate with databases, cloud services, Kubernetes, Spark, dbt and HTTP/SQL endpoints, plus a full REST API. This image runs Airflow 3.2 on Python 3.12 with the LocalExecutor (single-node, no external Celery or Redis broker required), persisting DAG metadata, connections, variables and run history in a local PostgreSQL database. Apache-2.0 license, no vendor lock-in.

Why self-host Apache Airflow

Running Airflow on a VM you control keeps every DAG, connection string, credential and pipeline run history inside your own tenant rather than a hosted orchestration service. Self-hosting suits teams under GDPR, HIPAA or ISO 27001, and any workload where pipeline definitions, data-source secrets and execution logs must stay within your own perimeter with no per-task or per-seat fees.

What this VM image adds

Security hardening:

• api-server bound to loopback - the web UI and REST API listen on 127.0.0.1:8080 only; nginx terminates TLS on port 443 and reverse-proxies to it, so port 8080 is never exposed
• Per-instance secrets - the Fernet key (connection/variable encryption), the API secret key, the PostgreSQL password and the admin login are all generated uniquely at first boot and written to /root/airflow-credentials.txt (mode 0600); none are baked into the image
• PostgreSQL bound to loopback - the metadata database listens on 127.0.0.1 only and is never exposed externally
• nginx TLS reverse proxy on 443 - a self-signed certificate is generated at build, replaceable with your own CA-signed certificate (certbot pre-installed)
• Non-root service user - the scheduler, api-server, dag-processor and triggerer run as a non-root airflow user with UMask 0027
• Anonymous usage analytics disabled
• UFW firewall - TCP 443 for the web UI and REST API, TCP 22 for SSH; all other inbound dropped
• fail2ban - SSH brute-force protection
• AppArmor - mandatory access control
• CVE scan - every image is scanned with Trivy before release

OS hardening (CIS Level 1):

• CIS Ubuntu 24.04 LTS Level 1 Benchmark via ansible-lockdown
• auditd - system call auditing for critical paths
• SSH hardening - PasswordAuthentication disabled, key-only access, PermitRootLogin no, LoginGraceTime 60
• Kernel hardening - SYN cookies, ASLR, rp_filter, IPv6 off
• /tmp as tmpfs - nosuid, nodev, noexec
• Azure IMDS endpoints - egress rules pre-configured (169.254.169.254, 168.63.129.16)

Compliance artifacts (inside the VM):

• SBOM - CycloneDX 1.6 at /etc/lynxroute/sbom.json
• CIS Conformance Report at /etc/lynxroute/cis-report.html (OpenSCAP, Azure tailoring profile, 0 FAIL rules)
• Tailored CIS profile at /usr/share/doc/lynxroute/CIS_TAILORED_PROFILE.md
• Operator credentials file at /root/airflow-credentials.txt (mode 0600) with the admin login and per-instance secrets

Quick Start

1. Deploy VM from Azure Marketplace (Standard_D2s_v3 or larger recommended)
2. Open NSG: TCP 443 from your trusted sources, TCP 22 from your management IPs only
3. SSH: ssh -i key.pem azureuser@<PUBLIC_IP>, then sudo cat /root/airflow-credentials.txt for the admin password
4. Open https://<PUBLIC_IP>/, accept the self-signed certificate warning, and log in as admin with the generated password
5. The DAGs view lists bundled example DAGs (paused by default). Unpause one and trigger it to watch the scheduler run tasks; drop your own DAG files into /opt/airflow/dags
6. For production, point a domain at the instance and run sudo certbot --nginx -d your.domain.com for a trusted certificate

Wait 1-3 minutes after first boot for the metadata database migration to complete before logging in. Restrict NSG port 443 to your own IP until you have changed the admin password. Built on Apache Airflow(R), a trademark of The Apache Software Foundation; an independently hardened build, not affiliated with or endorsed by the ASF.