# gtfs-dagster

Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB

## Input
This reads from the `config/agency_list.csv` file, edit this file to include the transit agencies that you want to scrape, add the relevant IDs from mobilitydatabase.org

## set your environment
### Data directory
edit dagster.yaml to specify the correct data directory under run_launcher.

Right now it says: `/home/ben/code/gtfs-dagster/data:/opt/dagster/app/data`, change the first part to where you want the data to be stored.

### .env file
copy `env.sample` to `.env` and change the database password and the mobility database refresh token

# Run it
`docker compose build`
`docker compose up -d`
access the Dagster web ui at 127.0.0.1:3001