# gtfs-dagster

Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB

## Input
You define which agencies and feeds to scrape with the file`config/agency_list.csv`

To include the transit agencies that you want to scrape, add the relevant IDs from mobilitydatabase.org

See `config/agency_list.csv.sample` for an example.

## set your environment

### .env file
copy `env.sample` to `.env` and change:
- Postgres database password - make it something random before the first run
- MobilityDatabase.org API token
- Location of data, config, and postgres_data directories (default is in working directory)



# Run it
`docker compose build`
`docker compose up -d`
access the Dagster web ui at 127.0.0.1:3001