gtfs-dagster/README.md

gtfs-dagster

Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB

Input

You define which agencies and feeds to scrape with the fileconfig/agency_list.csv

To include the transit agencies that you want to scrape, add the relevant IDs from mobilitydatabase.org

See config/agency_list.csv.sample for an example.

set your environment

.env file

copy env.sample to .env and change:

• Postgres database password - make it something random before the first run
• MobilityDatabase.org API token
• Location of data, config, and postgres_data directories (default is in working directory)

Run it

docker compose build docker compose up -d access the Dagster web ui at 127.0.0.1:3001