Orchestration README

The orchestration module wraps the serving tutorial’s local data platform in Dagster. The code location lives at tutorial_orchestration.definitions.

The runnable rep

Rep #8 gives you a run-once-and-read view of the conductor — no UI server needed:

make -C tutorial/orchestration run        # print the topology (assets, jobs, schedule, sensors)
make -C tutorial/orchestration check      # the topology + partition-materialization suite
make -C tutorial/orchestration clean      # remove pytest caches
make -C tutorial/orchestration dagster-dev  # the full Dagster UI (long-running, optional)

show_topology.py loads the Definitions object and prints the 31-asset graph: the dlt publisher loads, every dbt staging/model asset, the embeddings, accountability, and evaluation assets — plus the two asset checks on stg_unified_impressions, the three jobs, the daily schedule, and the two sensors.

The checkpoint, worked

Which sensor kicks off raw ingestion when a new source file lands, and what does the daily schedule materialise without any file event at all?

The raw_download_sensor watches the raw landing area and requests one raw_download_ingestion_job run per new file — event-driven ingestion. The daily_serving_platform_schedule fires on the clock, materialising the serving_platform_all_assets_job (the whole 31-asset graph) for the day’s Copenhagen impression-date partition, regardless of whether any file arrived. One graph, two triggers: a sensor for “data showed up” and a schedule for “it’s time” — which is the whole point of an orchestrator over a pile of cron jobs.

What makes this a good rep

Following the foundations template (ADR-0033): make run loads the real Definitions and prints the topology hermetically (no server, no data), make check runs the 11-test suite (topology wiring + a real single-partition materialization over a 1k fixture), and make clean clears caches. The full UI stays one make dagster-dev away.

To open the full Dagster UI:

make -C tutorial/orchestration dagster-dev

The asset graph starts with three publisher-level dlt assets: raw/ebnerd, raw/adressa, and raw/mind. The dbt project is exposed through dagster-dbt, so every current staging, modeling, and editorial dbt model is visible as a Dagster asset. The staging models depend on their publisher’s raw asset, which makes downstream materialization pull the required ingestion work into the run.

stg_unified_impressions carries two Dagster asset checks: row_count and freshness. The daily schedule targets the full asset graph, and the raw_download_sensor watches data/raw-download/ for newly downloaded source files before triggering raw ingestion.