diff --git a/README.md b/README.md index 61ee057..97b15cc 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,20 @@ # gtfs-dagster -Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB \ No newline at end of file +Dagster setup that scrapes GTFS and GTFS-RT for specified transit agencies and adds them to a DuckDB + +## Input +This reads from the `data/gtfs/agency_list.csv` file, edit this file to include the transit agencies that you want to scrape, add the relevant IDs from mobilitydatabase.org + +## set to your environment +### Data directory +edit dagster.yaml to specify the correct data directory under run_launcher. + +Right now it says: `/home/ben/code/gtfs-dagster/data:/opt/dagster/app/data`, change the first part to where you want the data to be stored. + +### .env file +copy `env.sample` to `.env` and change the password + +# Run it +`docker compose build` +`docker compose up -d` +access the Dagster web ui at 127.0.0.1:3001 diff --git a/env.sample b/env.sample new file mode 100644 index 0000000..f06225f --- /dev/null +++ b/env.sample @@ -0,0 +1,3 @@ +POSTGRES_USER=dagster +POSTGRES_PASSWORD=change-me +POSTGRES_DB=dagster