# gtfs-dagster

Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB

## Quick start

1. Edit the .env file.
copy `env.sample` to `.env` and change:
  - Postgres database password - make it something random before the first run
  - MobilityDatabase.org API token
  - Location of `data`, `config`, and `postgres_data` directories (default is in working directory). `config` is part of the repo as it comes with sample configuration files.

2. Edit `config/agency_list.csv`
  - See `config/agency_list.csv.sample` for an example.
  - Define which agencies and feeds to scrape with the file.
  - To include the transit agencies that you want to scrape, add the relevant Feed IDs from mobilitydatabase.org

3. Build the docker containers
`docker compose build`

4. Run the docker containers
`docker compose up -d`

5. Access the Dagster web ui at 127.0.0.1:3001

6. Materialize the first asset: `agency_list`

## To-do:
1. Change mobilitydata from using the API with a key, to using the csv on their GitHub page.
2. Load data into duckdb
3. Transform data in duckdb
4. Analyze data