Dagster: Open-source cloud-native data pipeline orchestrator

Dagster, an open-source orchestrator designed for managing data pipelines, facilitates the organization of data assets by determining their execution sequence and monitoring their interconnections. This is crucial for maintaining the integrity of data tables, datasets, machine learning models, and reports. When issues arise in a source table, Dagster's lineage graph helps identify affected downstream assets, providing a clear starting point for troubleshooting and repair.

In Dagster, data assets are defined as Python functions using the @dg.asset decorator, which outlines the dependencies between assets. The system reads these dependencies to construct a graph, visually represented in a web UI, aiding users in understanding the flow and connections of their data. The project includes examples such as linking country population data with models of continental changes to produce comprehensive continent statistics.

The orchestrator supports Python versions 3.9 through 3.14 and offers a library of integrations with popular data tools like dbt. Asset checks, which are data quality tests coded alongside the assets they assess, are integral to Dagster's functionality. These checks ensure data integrity by verifying conditions such as the absence of null values in columns. Users can import existing dbt tests into Dagster as asset checks, and the UI displays the status of each check, indicating whether it has passed, failed, or is pending.

Dagster also allows for the designation of blocking checks, which prevent the materialization of downstream assets if a check fails, thereby avoiding the propagation of errors. The orchestrator can automatically run assets and checks through schedules and sensors. Additionally, it supports partitioning, which divides an asset into segments like daily slices. When slices are missing or need updating, Dagster initiates a backfill process that can either run one operation per partition or consolidate them into a single operation, such as a Snowflake query.