Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB
| config | ||
| user_code | ||
| .gitignore | ||
| dagster.yaml | ||
| docker-compose.yaml | ||
| Dockerfile_dagster | ||
| Dockerfile_dagster_code | ||
| env.sample | ||
| LICENSE | ||
| README.md | ||
| workspace.yaml | ||
gtfs-dagster
Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB
Input
This reads from the config/agency_list.csv file, edit this file to include the transit agencies that you want to scrape, add the relevant IDs from mobilitydatabase.org
set your environment
Data directory
edit dagster.yaml to specify the correct data directory under run_launcher.
Right now it says: /home/ben/code/gtfs-dagster/data:/opt/dagster/app/data, change the first part to where you want the data to be stored.
.env file
copy env.sample to .env and change the database password and the mobility database refresh token
Run it
docker compose build
docker compose up -d
access the Dagster web ui at 127.0.0.1:3001