feat: pass updated seasons to catalog refresh endpoint#388
feat: pass updated seasons to catalog refresh endpoint#388
Conversation
📝 WalkthroughWalkthroughThe changes implement season-specific catalog refresh functionality. Changes
Sequence DiagramsequenceDiagram
actor User
participant MainScript as main.py
participant FileSystem as Data Directory
participant FerryWorkflow as ferry.yml
participant EvalWorkflow as evals.yml
participant CatalogAPI as Catalog Endpoints
User->>MainScript: Trigger crawl (crawl_classes/evals enabled)
MainScript->>MainScript: Process crawl data
MainScript->>FileSystem: Write seasons to ferry_updated_seasons.txt
User->>FerryWorkflow: Trigger ferry workflow
FerryWorkflow->>FileSystem: Check if ferry_updated_seasons.txt exists
alt File exists
FerryWorkflow->>FileSystem: Read SEASONS variable
FerryWorkflow->>CatalogAPI: POST /catalog/refresh?seasons=${SEASONS}
else File missing
FerryWorkflow->>CatalogAPI: POST /catalog/refresh (no seasons)
end
User->>EvalWorkflow: Trigger evals workflow with season input
EvalWorkflow->>CatalogAPI: POST /catalog/refresh?seasons=${{ inputs.season }}
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
.github/workflows/evals.yml (1)
88-89: Consider adding--failflag to detect HTTP errors.The curl commands will silently succeed even if the server returns a 4xx/5xx error. Adding
--fail(or-f) ensures the workflow step fails if the refresh endpoint returns an error status.- curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api.coursetable.com/api/catalog/refresh?seasons=${{ inputs.season }}" - curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api-staging.coursetable.com/api/catalog/refresh?seasons=${{ inputs.season }}" + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api.coursetable.com/api/catalog/refresh?seasons=${{ inputs.season }}" + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api-staging.coursetable.com/api/catalog/refresh?seasons=${{ inputs.season }}"main.py (1)
86-89: Potential for stale data if file persists from a previous run.The file
ferry_updated_seasons.txtis only written whencrawl_classesorcrawl_evalsis enabled. If a subsequent workflow run (e.g., transform-only) doesn't delete this file,ferry.ymlwould still read and use the stale seasons from a previous run.Consider either:
- Cleaning up the file after it's consumed in the workflow, or
- Writing an empty/sentinel file when crawl flags are not set to ensure the fallback path is taken.
This may be intentional if non-crawl runs don't trigger the "Regenerate static catalog" step, but worth confirming the workflow doesn't hit that step unexpectedly.
.github/workflows/ferry.yml (2)
123-126: Strip trailing newline from file content to avoid malformed URL.
catmay include a trailing newline if the file was written with one (Python'swrite_textdoesn't add one, but defensive coding is warranted). A trailing newline inSEASONSwould result in a malformed URL or unexpected behavior.if [ -f data/ferry_updated_seasons.txt ]; then - SEASONS=$(cat data/ferry_updated_seasons.txt) + SEASONS=$(tr -d '\n' < data/ferry_updated_seasons.txt) curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api.coursetable.com/api/catalog/refresh?seasons=${SEASONS}"
125-129: Consider adding--failflag to detect HTTP errors.Same as in
evals.yml, the curl commands will silently succeed even if the server returns an error. Adding--failensures the workflow fails visibly if the catalog refresh endpoint has issues.- curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api.coursetable.com/api/catalog/refresh?seasons=${SEASONS}" - curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api-staging.coursetable.com/api/catalog/refresh?seasons=${SEASONS}" + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api.coursetable.com/api/catalog/refresh?seasons=${SEASONS}" + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" "https://api-staging.coursetable.com/api/catalog/refresh?seasons=${SEASONS}" else - curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" https://api.coursetable.com/api/catalog/refresh - curl --silent --show-error -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" https://api-staging.coursetable.com/api/catalog/refresh + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" https://api.coursetable.com/api/catalog/refresh + curl --silent --show-error --fail -H "X-FERRY-SECRET: ${{secrets.FERRY_SECRET}}" https://api-staging.coursetable.com/api/catalog/refresh
1 participant
Summary by CodeRabbit