Dagster on Ubuntu 24.04 LTS

Dagster on Ubuntu 24.04 LTS by cloudimg

Dagster is the data orchestrator for the whole development lifecycle, built around data assets. Define your tables, models and reports as assets, and Dagster builds a lineage graph, runs and schedules them, and gives you a rich web UI to observe every run, asset materialisation, schedule and sensor. It is a leading choice for data engineers building observable, testable data platforms. The cloudimg image installs Dagster 1.13.10 in a dedicated Python virtualenv, runs the Dagster webserver and daemon as systemd services bound to loopback behind an nginx reverse proxy on port 80 with HTTP Basic auth, persists run and event storage on a dedicated Azure data disk, and generates a unique login password on the first boot of every VM. Backed by 24/7 expert support.

Dedicated Data Disk

The Dagster instance (DAGSTER_HOME) - the SQLite run, event and schedule storage - lives on a dedicated, independently resizable Azure data disk mounted at /var/lib/dagster, separate from the OS disk and re-provisioned with every VM. Snapshot it to back up your run history.

Secure By Default

Dagster has no built-in authentication, so the webserver is bound to loopback and nginx fronts it on port 80 with HTTP Basic auth. The per-VM admin password is unique to each VM, generated on first boot and stored in a root-only file. Front the proxy with TLS for production.

Why Choose cloudimg?

• 24/7 Expert Support with guaranteed 24 hour response. Contact support@cloudimg.co.uk
• Production Ready from Launch Pre configured, security patched, and validated before publication
• Azure Native Integration Built with Azure Linux Agent, cloud init, and Gen2 Hyper V

What is Included

• Dagster 1.13.10 with the webserver and daemon in a Python virtualenv at /opt/dagster/venv
• A sample asset project at /opt/dagster/project
• HTTP Basic auth with a per-VM password in a root-only file
• A dedicated Azure data disk at /var/lib/dagster for run and event storage
• dagster-webserver.service, dagster-daemon.service and nginx.service as systemd units

Use Cases

Data asset orchestration and lineage, ETL and ML pipeline scheduling, data quality and observability, a self-hosted control plane for data platforms, and a self-hosted alternative to managed orchestration services.

Visit www.cloudimg.co.uk/guides/dagster-on-ubuntu-24-04-azure for the full user guide.

All product and company names are trademarks or registered trademarks of their respective holders. Use of them does not imply any affiliation with or endorsement by them.