diff --git a/.gitignore b/.gitignore index ef25eee..eb5b7a6 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,10 @@ __pycache__/ *.pyc +*.egg-info/ +.venv/ +venv/ +dist/ .claude/settings.local.json .mcp.json +.env* +.DS_Store diff --git a/CLAUDE.md b/CLAUDE.md index 6aa98ca..eb9a7f5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,9 +137,14 @@ When contributing to this repository: ## Active Technologies - Python >=3.13 (constraint from all `pyproject.toml` files) + `requests>=2.31` (shared client), `mcp>=1.0` (MCP server only) (003-generalize-mcp-client) - `action_log.jsonl` (append-only, MCP layer only) (003-generalize-mcp-client) +- Python >=3.13 + `requests>=2.31`, `github-projects-client` (local editable) (004-or-filter-syntax) +- N/A (stateless — reads from GitHub API, writes TSV) (004-or-filter-syntax) --- *This document should be updated as the repository evolves and new conventions are established.* + +## Recent Changes +- 004-or-filter-syntax: Added Python >=3.13 + `requests>=2.31`, `github-projects-client` (local editable) diff --git a/github-project-export/README.md b/github-project-export/README.md index 36c56e8..cb906fa 100644 --- a/github-project-export/README.md +++ b/github-project-export/README.md @@ -44,8 +44,8 @@ Exit codes: `0` success (including zero matching items → header-only TSV), `1` | Key | Required | Description | |-----|----------|-------------| | `projectUrl` | Yes | Org project URL: `https://github.com/orgs/ORG/projects/N` | -| `query` | No* | Single project filter string (`q`). If both `query` and `queryParts` exist, **non-empty `query` wins**. | -| `queryParts` | No* | Array of strings; joined with spaces to form `q` when `query` is not used (or is empty). **Every element must be a string.** | +| `query` | No* | Single project filter string (`q`). Supports OR syntax (see below). If both `query` and `queryParts` exist, **non-empty `query` wins**. | +| `queryParts` | No* | Array of strings; joined with spaces to form `q` when `query` is not used (or is empty). OR syntax applies after joining. **Every element must be a string.** | | `fields` | Yes | Non-empty array of column headers. Order = TSV column order. | | `outputFile` | No | `null` or omit → stdout; non-empty string → file path. **`""` is invalid.** | @@ -89,10 +89,76 @@ Values come from the linked **issue or pull request** (`content`), not from cust You still get a **TSV header row** and **no data rows**. +### OR syntax + +Combine multiple filter branches with `OR`. Terms before the first parenthesized group form a **shared prefix** applied to every branch. Each branch becomes a separate API query; results are union-merged with deduplication. + +``` +shared-prefix (branch-1) OR (branch-2) OR (branch-3) +``` + +- **Shared prefix**: terms before the first `(` — applied to every branch +- **Branches**: each `(...)` group contains branch-specific terms +- **OR**: uppercase keyword separating groups +- **Backward compatible**: queries without `OR` or parentheses work unchanged + +**Example** — unassigned items from one milestone, assigned items from another (each branch has different conditions that can't be expressed in a single query): + +```json +{ + "query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)" +} +``` + +This expands to two queries: +1. `is:issue milestone:"M4.0: mainnet staged" no:assignee` +2. `is:issue milestone:"M4.1: mainnet ready" has:assignee` + +Items appearing in both branches are deduplicated by ID. + +**Real-world use case** — all done issues plus recently updated issues (not suitable for golden files since the board changes, but the most common use): + +```json +{ + "query": "is:issue (status:\"🎉 Done\") OR (-last-updated:7days)" +} +``` + +#### Limitations + +This is single-level OR expansion, not full boolean search. Each OR branch becomes a separate API query whose results are unioned. It does **not** support the nested boolean grouping that GitHub Issues search and Lucene/Elasticsearch offer. + +**Not supported:** + +``` +# Nested parentheses +((milestone:"M4.0" OR milestone:"M4.1") AND status:"🎉 Done") +→ Error: Nested parentheses are not supported + +# Filter terms after the last group +(status:A) OR (status:B) is:issue +→ Error: Filter terms after the last group are not allowed + +# OR without parenthesized groups +is:issue OR is:pr +→ Error: OR requires parenthesized groups + +# AND keyword (use the shared prefix instead) +(milestone:"M4.0") AND (status:"🎉 Done") +→ Not recognized — AND is not a keyword; write: status:"🎉 Done" (milestone:"M4.0") + +# OR inside a single group +(milestone:"M4.0" OR milestone:"M4.1") +→ Error: OR inside parentheses is not supported +``` + +The shared-prefix model covers the most common real-world cases (2–5 branches with shared context). For queries that need multi-dimensional boolean grouping, run separate exports. + ### Examples - [examples/export.example1.json](examples/export.example1.json) — narrow filter, matches the live integration fixture. - [examples/export.example2.json](examples/export.example2.json) — broader columns (milestone, assignees, reviewers, etc.). +- [examples/export.example3.json](examples/export.example3.json) — OR query with different conditions per branch. ## Implementation notes diff --git a/github-project-export/examples/export.example3.json b/github-project-export/examples/export.example3.json new file mode 100644 index 0000000..cf75a7b --- /dev/null +++ b/github-project-export/examples/export.example3.json @@ -0,0 +1,14 @@ +{ + "projectUrl": "https://github.com/orgs/FilOzone/projects/14", + "query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)", + "fields": [ + "Repository", + "Id", + "Title", + "Status", + "Milestone", + "Assignees", + "url" + ], + "outputFile": null +} diff --git a/github-project-export/github_project_export/config_schema.py b/github-project-export/github_project_export/config_schema.py index 6eaa5ed..401acee 100644 --- a/github-project-export/github_project_export/config_schema.py +++ b/github-project-export/github_project_export/config_schema.py @@ -7,6 +7,8 @@ from pathlib import Path from typing import Any, List, Optional +from github_projects_client import expand_or_query + from github_project_export.board_url import parse_org_project_url @@ -43,6 +45,10 @@ def _require_non_empty_query(data: dict[str, Any]) -> str: q_chars = q.strip() if isinstance(q, str) else "" if q_chars: + try: + expand_or_query(q_chars) + except ValueError as e: + raise ConfigError(str(e)) from e return q_chars if qp is not None and len(qp) > 0: @@ -51,6 +57,10 @@ def _require_non_empty_query(data: dict[str, Any]) -> str: raise ConfigError( "'query' is empty and 'queryParts' produces an empty filter", ) + try: + expand_or_query(joined) + except ValueError as e: + raise ConfigError(str(e)) from e return joined raise ConfigError( diff --git a/github-project-export/github_project_export/rest_export.py b/github-project-export/github_project_export/rest_export.py index 78dbfbf..520890c 100644 --- a/github-project-export/github_project_export/rest_export.py +++ b/github-project-export/github_project_export/rest_export.py @@ -8,6 +8,7 @@ import requests from github_projects_client import ( + expand_or_query, fetch_items_rest as fetch_project_v2_items_rest, list_field_ids_by_name as list_project_v2_field_ids_by_name, ) @@ -187,12 +188,32 @@ def export_rows( ) columns, field_ids = build_columns(fields, board_map) - result = fetch_project_v2_items_rest( - session, - org=org, - project_number=project_number, - query=query, - field_ids=field_ids if field_ids else None, - ) + queries = expand_or_query(query) - return [_item_to_row(it, columns) for it in result["items"]] + if len(queries) == 1: + result = fetch_project_v2_items_rest( + session, + org=org, + project_number=project_number, + query=queries[0], + field_ids=field_ids if field_ids else None, + ) + raw_items = result["items"] + else: + raw_items: List[Dict[str, Any]] = [] + seen_ids: set[int] = set() + for q in queries: + result = fetch_project_v2_items_rest( + session, + org=org, + project_number=project_number, + query=q, + field_ids=field_ids if field_ids else None, + ) + for item in result["items"]: + item_id = item.get("id") + if item_id not in seen_ids: + seen_ids.add(item_id) + raw_items.append(item) + + return [_item_to_row(it, columns) for it in raw_items] diff --git a/github-project-export/tests/fixtures/fixture_1_output.tsv b/github-project-export/tests/fixtures/fixture_1_output.tsv index 18b1bb9..4f7a5d9 100644 --- a/github-project-export/tests/fixtures/fixture_1_output.tsv +++ b/github-project-export/tests/fixtures/fixture_1_output.tsv @@ -1,4 +1,5 @@ Repository Id Title Status url -dealbot 411 Investigate staging Dealbot startup failure 🎉 Done https://github.com/FilOzone/dealbot/issues/411 -filecoin-services 411 Cleanup Calibration datasets to (temporarily) reduce chain bandwidth pressure before GA 🎉 Done https://github.com/FilOzone/filecoin-services/issues/411 synapse-sdk 411 fix: error outputs out of lotus are weird 🎉 Done https://github.com/FilOzone/synapse-sdk/pull/411 +filecoin-services 411 Cleanup Calibration datasets to (temporarily) reduce chain bandwidth pressure before GA 🎉 Done https://github.com/FilOzone/filecoin-services/issues/411 +dealbot 411 Investigate staging Dealbot startup failure 🎉 Done https://github.com/FilOzone/dealbot/issues/411 +filecoin-pin 411 filecoin-pin-upload action failing due to pnpm 🎉 Done https://github.com/filecoin-project/filecoin-pin/issues/411 diff --git a/github-project-export/tests/fixtures/fixture_2_input.json b/github-project-export/tests/fixtures/fixture_2_input.json new file mode 100644 index 0000000..cf75a7b --- /dev/null +++ b/github-project-export/tests/fixtures/fixture_2_input.json @@ -0,0 +1,14 @@ +{ + "projectUrl": "https://github.com/orgs/FilOzone/projects/14", + "query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)", + "fields": [ + "Repository", + "Id", + "Title", + "Status", + "Milestone", + "Assignees", + "url" + ], + "outputFile": null +} diff --git a/github-project-export/tests/fixtures/fixture_2_output.tsv b/github-project-export/tests/fixtures/fixture_2_output.tsv new file mode 100644 index 0000000..f8d6564 --- /dev/null +++ b/github-project-export/tests/fixtures/fixture_2_output.tsv @@ -0,0 +1,241 @@ +Repository Id Title Status Milestone Assignees url +synapse-sdk 564 Cut Synapse-SDK 0.37.0 🎉 Done M4.0: mainnet staged https://github.com/FilOzone/synapse-sdk/issues/564 +dealbot 73 Deploy Dealbot to Mainnet 🎉 Done M4.0: mainnet staged https://github.com/FilOzone/dealbot/issues/73 +filecoin-services 379 [Tracker] Mainnet is staged for GA 🎉 Done M4.0: mainnet staged https://github.com/FilOzone/filecoin-services/issues/379 +synapse-sdk 576 StorageManagerUploadOptions is not exported from @filoz/synapse-sdk 🎉 Done M4.0: mainnet staged https://github.com/FilOzone/synapse-sdk/issues/576 +synapse-sdk 591 Cut Synapse-SDK v0.38.0 🎉 Done M4.0: mainnet staged https://github.com/FilOzone/synapse-sdk/issues/591 +filecoin-pin 324 Failure after upgrade to filecoin-pin@0.15.1 🎉 Done M4.0: mainnet staged https://github.com/filecoin-project/filecoin-pin/issues/324 +dealbot 159 "GA ""Retrieval"" check implementation and documentation" 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/159 +synapse-sdk 483 Review/update API for GA 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/483 +synapse-sdk 484 GA: API Changes 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/484 +synapse-sdk 563 Migration Guide / Docs - Mainnet Ready Synapse Release 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/563 +synapse-sdk 242 feat: support safer size and rate/lockup calculations 🎉 Done M4.1: mainnet ready lordshashank https://github.com/FilOzone/synapse-sdk/issues/242 +synapse-sdk 308 Pre-flight checks before upload 🎉 Done M4.1: mainnet ready lordshashank https://github.com/FilOzone/synapse-sdk/issues/308 +tpm-utils 4 FOG WG notifier of open PRs 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/tpm-utils/issues/4 +filecoin-services 370 Runbooks for common support/oncall actions 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/370 +filecoin-services 359 Share script for how filecoinpin.contact can get the peerIds of all SPs [team/filecoin-pin] 🎉 Done M4.1: mainnet ready bajtos https://github.com/FilOzone/filecoin-services/issues/359 +dealbot 70 Consolidate dealbot-infra with other infra 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/70 +dealbot 77 Make operational readiness plan 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/77 +dealbot 86 [Op Readiness: P0]: publish frontend and backend docker images to github container registry 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/86 +dealbot 111 Robust wallet management 🎉 Done M4.1: mainnet ready rjan90,rvagg,SgtPooki,silent-cipher https://github.com/FilOzone/dealbot/issues/111 +dealbot 87 [Op Readiness: P0] Export prometheus metrics 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/87 +dealbot 90 [Op Readiness: P0] Runbook and ops docs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/90 +dealbot 102 fix: hardcode a version number for dataSet metadata 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/102 +foc-devnet 10 Local-net: Make initial/GA project plan 🎉 Done M4.1: mainnet ready BigLep,redpanda-f https://github.com/FilOzone/foc-devnet/issues/10 +filecoin-services 346 Document FWSS upgrade process 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/346 +synapse-sdk 540 Decision: Endorsement Certificates vs Smart Contract Approach 🎉 Done M4.1: mainnet ready wjmelements https://github.com/FilOzone/synapse-sdk/issues/540 +filecoin-pin 8 Add `rm --piece $pieceCid` command 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/filecoin-project/filecoin-pin/issues/8 +filecoin-services 229 Proposal: monorepo & no git submodules 🎉 Done M4.1: mainnet ready redpanda-f,rjan90 https://github.com/FilOzone/filecoin-services/issues/229 +filecoin-services 212 Define release strategy for FilecoinWarmStorageServices 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/212 +filecoin-services 31 Debt Handling and Payment Failure Policy Design 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/FilOzone/filecoin-services/issues/31 +filecoin-services 55 Document Pricing / Top-up & Renewal in a spec.md 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/55 +filecoin-services 178 Add CI for subgraph 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-services/issues/178 +synapse-sdk 51 Streaming PDP piece upload flow 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/synapse-sdk/issues/51 +filecoin-services 211 unpaginated getApprovedProviders 🎉 Done M4.1: mainnet ready Chaitu-Tatipamula https://github.com/FilOzone/filecoin-services/issues/211 +synapse-sdk 203 Payments operatorApprovals doesnt fail if wallet is not connect 🎉 Done M4.1: mainnet ready DarkLord017 https://github.com/FilOzone/synapse-sdk/issues/203 +synapse-sdk 204 Port all tests to the new rpc mock setup 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/204 +synapse-sdk 232 Stop exporting all the things on the top level 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/232 +filecoin-pay 235 Adopt Contract Name and File Name Versioning 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-pay/issues/235 +fs-pm 240 As the protocol, I want to have a content routing solution for IPFS CIDs included in PDP deals that doesn't rely on cid.contact during its season of instability and doesn't require manual Kubo config updates 🎉 Done M4.1: mainnet ready BigLep https://github.com/FilOzone/fs-pm/issues/240 +filecoin-pin-website 103 Load test filecoin-pin-website 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/filecoin-project/filecoin-pin-website/issues/103 +filecoin-services 250 2 copy of each upload 🎉 Done M4.1: mainnet ready wjmelements https://github.com/FilOzone/filecoin-services/issues/250 +synapse-sdk 259 Test new faucet api 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/259 +curio 669 pdpv0: Delete piece locally after a client initiated deletion 🎉 Done M4.1: mainnet ready Kubuxu https://github.com/filecoin-project/curio/issues/669 +filecoin-services 254 M3.5 Release checklist 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/254 +synapse-sdk 225 Remove ethers from dependencies 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/225 +synapse-sdk 503 extract testing rpc mock to its own package 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/503 +synapse-sdk 273 use http/ws client for reads instead of wallet client (fix metamask) 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/273 +pdp 216 Update deployment parameters in the docs/comments/ 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/pdp/issues/216 +curio 688 M3.1 Curio PDP Burndown Overview 🎉 Done M4.1: mainnet ready rjan90 https://github.com/filecoin-project/curio/issues/688 +synapse-sdk 296 Bug: pieceStatus confuses piece ownership 🎉 Done M4.1: mainnet ready juliangruber https://github.com/FilOzone/synapse-sdk/issues/296 +synapse-sdk 312 Default Replication 🎉 Done M4.1: mainnet ready wjmelements https://github.com/FilOzone/synapse-sdk/issues/312 +synapse-sdk 321 Filecoin Onchain Cloud docs 🎉 Done M4.1: mainnet ready longfeiWan9 https://github.com/FilOzone/synapse-sdk/issues/321 +synapse-sdk 322 Docs: Introduction 🎉 Done M4.1: mainnet ready longfeiWan9 https://github.com/FilOzone/synapse-sdk/issues/322 +synapse-sdk 323 Docs: Getting started 🎉 Done M4.1: mainnet ready longfeiWan9 https://github.com/FilOzone/synapse-sdk/issues/323 +curio 713 pdpv0: implement EXTRA_DATA_MAX_SIZE limit 🎉 Done M4.1: mainnet ready siddharthbaleja7 https://github.com/filecoin-project/curio/issues/713 +synapse-sdk 221 Create getCosts method 🎉 Done M4.1: mainnet ready lordshashank https://github.com/FilOzone/synapse-sdk/issues/221 +synapse-sdk 345 Docs: concepts section 🎉 Done M4.1: mainnet ready longfeiWan9 https://github.com/FilOzone/synapse-sdk/issues/345 +synapse-sdk 346 Docs: Developer/Synapse-sdk 🎉 Done M4.1: mainnet ready nijoe1 https://github.com/FilOzone/synapse-sdk/issues/346 +pdp 230 Transfer ownership of PDP contracts to SAFE msig 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/pdp/issues/230 +synapse-sdk 358 """dev"" selector should use `serviceStatus=dev` in SP capabilities list" 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/synapse-sdk/issues/358 +synapse-sdk 359 Increase TRANSACTION_PROPAGATION_TIMEOUT_MS 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/synapse-sdk/issues/359 +synapse-sdk 360 New Costs API 🎉 Done M4.1: mainnet ready lordshashank https://github.com/FilOzone/synapse-sdk/issues/360 +dealbot 32 Add SP Curio version to Dealbot page 🎉 Done M4.1: mainnet ready pali101 https://github.com/FilOzone/dealbot/issues/32 +curio 758 ServiceProviderRegistry: align on canonical decoding `address` from `bytes` 🎉 Done M4.1: mainnet ready siddharthbaleja7 https://github.com/filecoin-project/curio/issues/758 +curio 760 pdpv0: deletion in pdp proving task is too eager 🎉 Done M4.1: mainnet ready LexLuthr https://github.com/filecoin-project/curio/issues/760 +filecoin-pin-website 31 fix: do not call to synapse-sdk directly 🎉 Done M4.1: mainnet ready juliangruber https://github.com/filecoin-project/filecoin-pin-website/issues/31 +synapse-sdk 363 Change client-side telemetry in preparation for GA 🎉 Done M4.1: mainnet ready juliangruber https://github.com/FilOzone/synapse-sdk/issues/363 +filecoin-pin 65 feat: support multiple providers 🎉 Done M4.1: mainnet ready rvagg https://github.com/filecoin-project/filecoin-pin/issues/65 +filecoin-pin 189 Add `data-set $dataSetId --terminate` 🎉 Done M4.1: mainnet ready akronim26 https://github.com/filecoin-project/filecoin-pin/issues/189 +curio 746 pdpv0: Optimize PDP Watch: 30-second delay between transaction confirmation in PDP vs onchain 🎉 Done M4.1: mainnet ready snadrus https://github.com/filecoin-project/curio/issues/746 +dealbot 44 7-day view in detailed metrics showing wrong count 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/44 +synapse-sdk 392 Synapse API to query the remaining egress allowances for a given data set 🎉 Done M4.1: mainnet ready pyropy https://github.com/FilOzone/synapse-sdk/issues/392 +filecoin-services 340 Transfer ownership of FWSS v1.0.0/v1.1.0 contracts to SAFE msig 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/340 +curio 786 pdp/ipni: IPNI assumes default HTTPS port when announcing 🎉 Done M4.1: mainnet ready beck-8,BigLep https://github.com/filecoin-project/curio/issues/786 +filecoin-services 347 Update contract links to Blockscout 🎉 Done M4.1: mainnet ready akronim26 https://github.com/FilOzone/filecoin-services/issues/347 +synapse-sdk 428 Update contract links to Blockscout 🎉 Done M4.1: mainnet ready akronim26 https://github.com/FilOzone/synapse-sdk/issues/428 +dealbot 55 Mainnet Dealbot Parameters 🎉 Done M4.1: mainnet ready BigLep https://github.com/FilOzone/dealbot/issues/55 +filecoin-services 351 validatePayerFunds for piecesAdded 🎉 Done M4.1: mainnet ready DarkLord017 https://github.com/FilOzone/filecoin-services/issues/351 +dealbot 59 Active Providers is showing too few provider 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/dealbot/issues/59 +dealbot 95 bug in UI for stats? 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/dealbot/issues/95 +dealbot 60 add auto-funding or alert mechanism for low balance in dealbot filecoin-pay account 🎉 Done M4.1: mainnet ready pali101 https://github.com/FilOzone/dealbot/issues/60 +filecoin-services 354 doc: Deployment Addresses json 🎉 Done M4.1: mainnet ready Chaitu-Tatipamula https://github.com/FilOzone/filecoin-services/issues/354 +dealbot 63 IPNI indexing: Average time to retrieve metrics is negative 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/63 +filecoin-pay-explorer 54 docs: need instructions for setup 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/54 +filecoin-pin-website 129 Add monitoring for the shared wallet balance 🎉 Done M4.1: mainnet ready rjan90 https://github.com/filecoin-project/filecoin-pin-website/issues/129 +dealbot 66 All dealbot deals are failing with `Timeout Errors During Deal Creation` 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/66 +filecoin-pin 268 Potential discrepancy in uploaded/stored accounting in pin 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/filecoin-project/filecoin-pin/issues/268 +synapse-sdk 468 Add type validation to docs 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/468 +filecoin-cloud 156 docs.filecoin.cloud plausible 🎉 Done M4.1: mainnet ready timfong888 https://github.com/FilOzone/filecoin-cloud/issues/156 +curio 815 pdpv0: cleanup functions may delete too much data 🎉 Done M4.1: mainnet ready LexLuthr https://github.com/filecoin-project/curio/issues/815 +dealbot 68 Set up 7 day database backups 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/68 +dealbot 69 [Op Readiness: P0] Filecoin-Pin integration 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/69 +curio 723 pdpv0: log IPNI announcement endpoints and their HTTP response 🎉 Done M4.1: mainnet ready beck-8 https://github.com/filecoin-project/curio/issues/723 +curio 814 pdpv0: panic in `processIndexingAndIPNICleanup` when creating IPNI removal advertisement 🎉 Done M4.1: mainnet ready LexLuthr https://github.com/filecoin-project/curio/issues/814 +pdp 235 `transfer-owner.sh` fails verification step when run in zsh 🎉 Done M4.1: mainnet ready akronim26 https://github.com/FilOzone/pdp/issues/235 +curio 821 pdpv0: sync from `main` 🎉 Done M4.1: mainnet ready snadrus https://github.com/filecoin-project/curio/issues/821 +filecoin-pay 253 Audit and documentation for auction 🎉 Done M4.1: mainnet ready rjan90,ZenGround0 https://github.com/FilOzone/filecoin-pay/issues/253 +filecoin-services 361 Document Migration Strategy for FWSS Contracts 🎉 Done M4.1: mainnet ready BigLep,rjan90 https://github.com/FilOzone/filecoin-services/issues/361 +foc-devnet 3 Make this repo public 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/3 +foc-devnet 5 Basic DevNet with End-to-End Deal + Retrieval 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/5 +foc-devnet 6 Run with Local/Branch Versions of Components 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/6 +foc-devnet 7 Example of how to use with Synapse 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/7 +foc-devnet 8 CI/Nightly End-to-End Validation of FOC as a whole 🎉 Done M4.1: mainnet ready galargh,redpanda-f https://github.com/FilOzone/foc-devnet/issues/8 +foc-devnet 9 Document for team adoption and maintenance 🎉 Done M4.1: mainnet ready BigLep,redpanda-f https://github.com/FilOzone/foc-devnet/issues/9 +synapse-sdk 469 Add support to mainnet in examples/cli 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/469 +filecoin-pay 255 Write up Auction System functionality in SPEC.md 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/FilOzone/filecoin-pay/issues/255 +synapse-sdk 486 Wrongly named `signerAddress` is misnamed in context selection & resolution methods 🎉 Done M4.1: mainnet ready akronim26 https://github.com/FilOzone/synapse-sdk/issues/486 +filecoin-pin 282 Add public PieceCID/CommP helper and top-level API for uploads 🎉 Done M4.1: mainnet ready bajtos https://github.com/filecoin-project/filecoin-pin/issues/282 +synapse-sdk 495 Simplify provider & data set resolution logic & API 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/synapse-sdk/issues/495 +dealbot 80 [SEV3] Dealbot stalled 🎉 Done M4.1: mainnet ready SgtPooki,silent-cipher https://github.com/FilOzone/dealbot/issues/80 +dealbot 85 [Op Readiness: P0] separate web ui and dealbot backend infrastructure 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/85 +synapse-sdk 504 figure out public/wallet client options 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/504 +dealbot 93 [Op Readiness: P0] get filoz hosted domain and decide on permanent home for dealbot 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/93 +filecoin-pay-explorer 58 Weekly unique query discrepancy 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/58 +filecoin-pay-explorer 59 Integrate Plausible Analytics for usage tracking 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/59 +filecoin-pay-explorer 63 Provide subgraph endpoint access for settlement and wallet metrics 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-pay-explorer/issues/63 +filecoin-pay-explorer 64 Integrate numbers across public-facing dashboards 🎉 Done M4.1: mainnet ready davidgasquez,timfong888 https://github.com/FilOzone/filecoin-pay-explorer/issues/64 +dealbot 97 docs: clarify purpose and usage of dealbot env vars 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/97 +filecoin-pay-explorer 69 Link back to this repo so people can see the code, know where to file feature requests or bugs. 🎉 Done M4.1: mainnet ready pyropy https://github.com/FilOzone/filecoin-pay-explorer/issues/69 +synapse-sdk 437 Add intelligent synapse.payments.fund(amount) (improve golden path) 🎉 Done M4.1: mainnet ready lordshashank https://github.com/FilOzone/synapse-sdk/issues/437 +filecoin-services 371 Pre-GA review: Sybil fee cost burden and abuse vector 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/FilOzone/filecoin-services/issues/371 +synapse-sdk 520 Use getPDPConfig() instead of individual PDP config getters 🎉 Done M4.1: mainnet ready akronim26 https://github.com/FilOzone/synapse-sdk/issues/520 +filecoin-cloud 183 Add SP approval status to Service Table 🎉 Done M4.1: mainnet ready mirhamasala,timfong888 https://github.com/FilOzone/filecoin-cloud/issues/183 +filecoin-cloud 184 Rethink refresh functionality on Service Table 🎉 Done M4.1: mainnet ready CharlyMartin https://github.com/FilOzone/filecoin-cloud/issues/184 +foc-devnet 15 Build script timing causes compilation failure for embedded MockUSDFC archive 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/15 +dealbot 113 bug: The retrieval task cannot be ended 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/113 +filecoin-services 375 Rail settlement check on Dataset Deletion 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/filecoin-services/issues/375 +pdp 240 Update gas benchmarks with production stack 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/FilOzone/pdp/issues/240 +curio 860 PDP(v0): submitting too-late proofs `ProvingPeriodPassed` 🎉 Done M4.1: mainnet ready beck-8 https://github.com/filecoin-project/curio/issues/860 +foc-devnet 17 Nightly CI runs, needs clarity 🎉 Done M4.1: mainnet ready BigLep,redpanda-f https://github.com/FilOzone/foc-devnet/issues/17 +foc-devnet 23 Run fails on proof param download 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/23 +infra 2 Epic: get basic infra set up 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/2 +infra 3 set up argocd 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/3 +infra 4 set up traefik 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/4 +infra 5 set up basic prod cluster and ensure all are happy with setup 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/5 +infra 6 set up staging cluster 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/6 +infra 7 feat: deploy dealbot on filoz infra 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/7 +infra 8 Create DNS entry pointing to dealbot staging and prod 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/8 +infra 15 ensure betterstack is setup for dealbot logs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/15 +curio 870 pdpv0: ServiceProviderRegistry capability entry IPNIPeerID isn't consistently cased 🎉 Done M4.1: mainnet ready rvagg https://github.com/filecoin-project/curio/issues/870 +filecoin-services 382 Endorsed Providers Contract 🎉 Done M4.1: mainnet ready wjmelements https://github.com/FilOzone/filecoin-services/issues/382 +filecoin-services 383 `terminateService` and `terminateCDNService` revert if CDN rails already terminated externally 🎉 Done M4.1: mainnet ready pyropy https://github.com/FilOzone/filecoin-services/issues/383 +tpm-utils 6 Create FOC monitoring and incident runbook for FOC websites 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/tpm-utils/issues/6 +dealbot 119 Change task scheduling to use startup-relative intervals instead of wall-clock time 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/119 +curio 880 pdpv0: Task backlog will affect normal data uploading 🎉 Done M4.1: mainnet ready beck-8 https://github.com/filecoin-project/curio/issues/880 +filecoin-services 391 Runbook for how to get a dump of relevant BetterStack logs 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/391 +dealbot 123 Determine GA scope 🎉 Done M4.1: mainnet ready BigLep https://github.com/FilOzone/dealbot/issues/123 +infra 23 "ensure that merging of ""merge to prod"" results in deployment to prod dealbot" 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/23 +foc-devnet 29 `cargo run -- init` fails with NotFound when docker is not installed. 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/29 +foc-devnet 30 `cargo run -- init` fails in `groupadd` with GID already exists 🎉 Done M4.1: mainnet ready beck-8 https://github.com/FilOzone/foc-devnet/issues/30 +curio 892 pdpv0: understand load capacity at GA 🎉 Done M4.1: mainnet ready beck-8 https://github.com/filecoin-project/curio/issues/892 +filecoin-services 392 Overall FOC Operational Readiness Review 🎉 Done M4.1: mainnet ready BigLep https://github.com/FilOzone/filecoin-services/issues/392 +synapse-sdk 557 Bug: piece deletion not working with synapse 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/557 +infra 27 solve errors in logs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/27 +infra 28 kaggle 403 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/28 +infra 29 dataset folder error 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/29 +foc-devnet 35 Provide `aarch64` / MacOS M series chip support for `foc-devnet` 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/35 +foc-devnet 36 Canonicalize `foc-devnet` SetupContext, ContractAddresses and Wallet keys 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/36 +foc-devnet 37 Fix issues with `latest` symlinks 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/37 +foc-devnet 38 Make `~/.foc-devnet` directory configurable. 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/38 +infra 32 create dealbot deployment and management docs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/32 +dealbot 139 Size effort for off-chain PDP challenges 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/139 +infra 33 make sure prod dealbot is testing unapproved providers as well 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/infra/issues/33 +dealbot 144 bug: when we retrieve or verify content downloaded from SP or IPFS, we do not validate the actual content 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/144 +filecoin-services 398 "Recoverability test: ensure realistic SP outages are recoverable ""quick enough""" 🎉 Done M4.1: mainnet ready TippyFlitsUK https://github.com/FilOzone/filecoin-services/issues/398 +curio 904 pdpv0: PDP subsystem documentation 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/filecoin-project/curio/issues/904 +dealbot 145 handle aborted upload error properly 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/145 +filecoin-pin 309 bug: payments setup doesnt respect `--mainnet` 🎉 Done M4.1: mainnet ready DeFiPrince https://github.com/filecoin-project/filecoin-pin/issues/309 +curio 924 pdpv0: pdp_data_set_pieces.removed not updated after successful deletion transaction 🎉 Done M4.1: mainnet ready rjan90,ZenGround0 https://github.com/filecoin-project/curio/issues/924 +dealbot 158 "GA ""Data Storage"" check implementation and documentation" 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/158 +dealbot 160 "Document ""check"" events and metrics" 🎉 Done M4.1: mainnet ready BigLep,SgtPooki https://github.com/FilOzone/dealbot/issues/160 +filecoin-pay-explorer 89 Add entity for `OneTimePayment` 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/89 +dealbot 163 Routinely pause dealbot to allow for SP maintenance 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/163 +foc-devnet 48 Check for required binaries before allowing start 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/foc-devnet/issues/48 +dealbot 165 chainLatencyMs metric is misnamed or incorrectly calculated 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/165 +filecoin-pay-explorer 93 bug: rail lockup calculation error 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/93 +dealbot 168 fully remove use of SP_RECEIVED_RETRIEVE_REQUEST 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/dealbot/issues/168 +dealbot 169 remove ENABLE_CDN_TESTING and places that use it 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/169 +dealbot 170 Update to Synapse 0.38.0 (and corresponding FilecoinPin) 🎉 Done M4.1: mainnet ready juliangruber https://github.com/FilOzone/dealbot/issues/170 +dealbot 171 use only the random-unique-dataset creation method in datasource.service.ts 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/171 +dealbot 172 remove proxy_list and proxy_label 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/172 +filecoin-pay-explorer 95 "Add ""Total transacted per token"" metric for GA" 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/filecoin-pay-explorer/issues/95 +dealbot 173 stop trimming CID/spAddress/pieceCID in logs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/173 +dealbot 174 Document production configuration settings 🎉 Done M4.1: mainnet ready BigLep,geomatrick https://github.com/FilOzone/dealbot/issues/174 +dealbot 178 Run Data Storage check 4 times per hour per SP 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/178 +filecoin-services 407 Pull PDP submodule and add PDP_INIT_COUNTER param to PDPVerifier deployment scripts 🎉 Done M4.1: mainnet ready DarkLord017 https://github.com/FilOzone/filecoin-services/issues/407 +dealbot 209 We need to be able to view jobs 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/209 +filecoin-services 411 Cleanup Calibration datasets to (temporarily) reduce chain bandwidth pressure before GA 🎉 Done M4.1: mainnet ready beck-8,silent-cipher https://github.com/FilOzone/filecoin-services/issues/411 +filecoin-services 412 "Move ""subgraph"" to supported repository" 🎉 Done M4.1: mainnet ready nijoe1,rjan90 https://github.com/FilOzone/filecoin-services/issues/412 +synapse-sdk 589 foc-devnet Chain builder in synpase-core 🎉 Done M4.1: mainnet ready redpanda-f https://github.com/FilOzone/synapse-sdk/issues/589 +dealbot 222 Data retention fault metrics via subgraph 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/222 +dealbot 238 Determine what telemetry visualization/alarming stack we're using for GA: BetterStack or Grafana 🎉 Done M4.1: mainnet ready BigLep,rjan90,SgtPooki https://github.com/FilOzone/dealbot/issues/238 +dealbot 239 Simplified dealbot functionality for GA 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/239 +synapse-sdk 599 Require `source` parameter for SDK initialisation for namespace isolation 🎉 Done M4.1: mainnet ready hugomrdias,rvagg https://github.com/FilOzone/synapse-sdk/issues/599 +filecoin-services 418 FWSS M4.1 Contract Upgrade 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/418 +dealbot 245 "Document job ""design/architecture""" 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/245 +filecoin-services 422 Adjust `challengeWindowSize` to 60 on FWSS Mainnet 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-services/issues/422 +infra 60 "add ""add to project"" github CI workflow to this repo" 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/infra/issues/60 +dealbot 258 we need to set deal/retrieval max timeout 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/258 +dealbot 260 pgboss/jobs mode should refresh providers daily 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/260 +synapse-sdk 602 Validate a terminating a already terminated dataset decodes the proper error with the recent lotus changes 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/602 +filecoin-cloud 232 Update roadmap 🎉 Done M4.1: mainnet ready jennijuju,mirhamasala,timfong888 https://github.com/FilOzone/filecoin-cloud/issues/232 +synapse-sdk 606 "addPieces fails with ""value.length"" error when metadata entry value is undefined" 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/synapse-sdk/issues/606 +synapse-sdk 613 Security: Validate operator address and allowance when custom operator is provided in approve operations 🎉 Done M4.1: mainnet ready Chaitu-Tatipamula https://github.com/FilOzone/synapse-sdk/issues/613 +dealbot 276 BetterStack: top-level dashboard summarizing SP performance as a whole 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/276 +dealbot 277 BetterStack: provider dashboard 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/277 +dealbot 278 BetterStack: dealbot logs can be publicly queried 🎉 Done M4.1: mainnet ready SgtPooki,silent-cipher https://github.com/FilOzone/dealbot/issues/278 +dealbot 279 Dealbot UI: basic table of SPs for easy links to dashboards 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/279 +dealbot 284 Dealbot ensures SPs-under-test have `MIN_NUM_DATASETS_FOR_CHECKS` datasets 🎉 Done M4.1: mainnet ready Chaitu-Tatipamula https://github.com/FilOzone/dealbot/issues/284 +filecoin-pin-website 146 Refresh session key 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/filecoin-project/filecoin-pin-website/issues/146 +dealbot 294 we should emit individual ttfb metrics for block-fetch strategy 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/294 +curio 1031 pdpv0: Sanity check for changing challenge window 🎉 Done M4.1: mainnet ready ZenGround0 https://github.com/filecoin-project/curio/issues/1031 +curio 1045 pdpv0: ResolveViewAddress can block indefinitely, hanging the chain scheduler 🎉 Done M4.1: mainnet ready rvagg https://github.com/filecoin-project/curio/issues/1045 +dealbot 310 fix: logs are structured and in one line. 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/310 +dealbot 311 fix: structured logs include providerId / providerAddress / pieceCid / ipfsRootCID where appropriate. 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/311 +filecoin-services 431 Event `ProviderRegistered` does not include name and description 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/filecoin-services/issues/431 +dealbot 319 Address Staging KubePodCrashLooping issue 🎉 Done M4.1: mainnet ready SgtPooki,silent-cipher https://github.com/FilOzone/dealbot/issues/319 +synapse-sdk 646 Update synapse-react to support multi sp upload flow 🎉 Done M4.1: mainnet ready hugomrdias https://github.com/FilOzone/synapse-sdk/issues/646 +filecoin-services 435 Options for the Sybil Fee Payment 🎉 Done M4.1: mainnet ready timfong888,ZenGround0 https://github.com/FilOzone/filecoin-services/issues/435 +dealbot 326 followup to data-retention checks 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/326 +dealbot 327 Review: Better Stack dashboard vs DealBot JTBD requirements 🎉 Done M4.1: mainnet ready BigLep,SgtPooki https://github.com/FilOzone/dealbot/issues/327 +synapse-sdk 652 Make FOC working group maintainer demos and tools more discoverable 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/synapse-sdk/issues/652 +tpm-utils 14 Improve discoverability of demo/example projects on FilOzone GitHub landing page 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/tpm-utils/issues/14 +filecoin-pin 348 Give option to skip content routing checks 🎉 Done M4.1: mainnet ready rvagg https://github.com/filecoin-project/filecoin-pin/issues/348 +synapse-sdk 657 Improve UploadResult ergonomics: add requestedCopies, complete, rename failures 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/synapse-sdk/issues/657 +dealbot 336 New UI intuitively sees storage/retrieval issues 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/336 +filecoin-pay-explorer 122 Setup goldsky payment 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/filecoin-pay-explorer/issues/122 +dealbot 342 Follow-up: make time window parsing DST/timezone-stable 🎉 Done M4.1: mainnet ready silent-cipher https://github.com/FilOzone/dealbot/issues/342 +dealbot 345 fixup: dealbot SP logs dashboard on betterstack 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/345 +curio 1088 pdpv0: settle payment rails every ~7 days 🎉 Done M4.1: mainnet ready LexLuthr https://github.com/filecoin-project/curio/issues/1088 +curio 1091 pdpv0: pdp/v1.3.0 for M4.1 🎉 Done M4.1: mainnet ready rjan90 https://github.com/filecoin-project/curio/issues/1091 +dealbot 349 review of dealbot on mainnet - 03-11-26 🎉 Done M4.1: mainnet ready timfong888 https://github.com/FilOzone/dealbot/issues/349 +filecoin-services 440 Add remaining endorsed SP certificates for M4.1 🎉 Done M4.1: mainnet ready rjan90,TippyFlitsUK https://github.com/FilOzone/filecoin-services/issues/440 +filecoin-services 443 Add pagination to clientDataSets and getClientDataSets 🎉 Done M4.1: mainnet ready rvagg https://github.com/FilOzone/filecoin-services/issues/443 +pdp 252 Calibnet: PDPVerifier upgrade from `v3.1.0` to `v3.2.0` 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/pdp/issues/252 +pdp 253 Mainnet: PDPVerifier upgrade from `v3.1.0` to `v3.2.0` 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/pdp/issues/253 +filecoin-pin 367 IPNI verification fails for SPs on Curio git_88428906 due to multiaddr format mismatch 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/filecoin-project/filecoin-pin/issues/367 +dealbot 382 Data storage checks failing for SPs on Curio git_88428906 🎉 Done M4.1: mainnet ready SgtPooki https://github.com/FilOzone/dealbot/issues/382 +curio 1110 pdpv0: advertise multiaddrs in ecosystem-compatible way 🎉 Done M4.1: mainnet ready beck-8 https://github.com/filecoin-project/curio/issues/1110 +infra 93 Configure dealbot to use dedicated RPC endpoint for better reliability 🎉 Done M4.1: mainnet ready rjan90 https://github.com/FilOzone/infra/issues/93 +dealbot 411 Investigate staging Dealbot startup failure 🎉 Done M4.1: mainnet ready rvagg,silent-cipher https://github.com/FilOzone/dealbot/issues/411 diff --git a/github-project-export/tests/test_config_validation.py b/github-project-export/tests/test_config_validation.py new file mode 100644 index 0000000..a33f4ff --- /dev/null +++ b/github-project-export/tests/test_config_validation.py @@ -0,0 +1,71 @@ +"""Tests for OR syntax validation at config load time (T012).""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from github_project_export.config_schema import ConfigError, load_export_config + + +def _write_config(tmp_path: Path, query: str) -> Path: + config = { + "projectUrl": "https://github.com/orgs/FilOzone/projects/14", + "query": query, + "fields": ["Title"], + } + p = tmp_path / "config.json" + p.write_text(json.dumps(config), encoding="utf-8") + return p + + +class TestOrSyntaxConfigValidation: + """Verify malformed OR queries raise ConfigError at config load time.""" + + def test_unmatched_open_paren(self, tmp_path: Path): + p = _write_config(tmp_path, "(a b") + with pytest.raises(ConfigError, match="Unmatched opening parenthesis"): + load_export_config(p) + + def test_or_without_parens(self, tmp_path: Path): + p = _write_config(tmp_path, "a OR b") + with pytest.raises(ConfigError, match="OR requires parenthesized groups"): + load_export_config(p) + + def test_nested_parens(self, tmp_path: Path): + p = _write_config(tmp_path, "((a)) OR (b)") + with pytest.raises(ConfigError, match="Nested parentheses"): + load_export_config(p) + + def test_empty_group(self, tmp_path: Path): + p = _write_config(tmp_path, "(a) OR ()") + with pytest.raises(ConfigError, match="Empty group"): + load_export_config(p) + + def test_trailing_terms(self, tmp_path: Path): + p = _write_config(tmp_path, "(a) OR (b) extra") + with pytest.raises(ConfigError, match="Filter terms after the last group"): + load_export_config(p) + + def test_valid_or_query_loads_ok(self, tmp_path: Path): + p = _write_config(tmp_path, "is:issue (status:A) OR (status:B)") + config = load_export_config(p) + assert config.query == "is:issue (status:A) OR (status:B)" + + def test_valid_plain_query_loads_ok(self, tmp_path: Path): + p = _write_config(tmp_path, "is:issue") + config = load_export_config(p) + assert config.query == "is:issue" + + def test_query_parts_or_validation(self, tmp_path: Path): + config = { + "projectUrl": "https://github.com/orgs/FilOzone/projects/14", + "queryParts": ["is:issue", "(a OR b)"], + "fields": ["Title"], + } + p = tmp_path / "config.json" + p.write_text(json.dumps(config), encoding="utf-8") + with pytest.raises(ConfigError, match="OR inside parentheses"): + load_export_config(p) diff --git a/github-project-export/tests/test_export_example_live.py b/github-project-export/tests/test_export_example_live.py index 4609a2c..b39d35d 100644 --- a/github-project-export/tests/test_export_example_live.py +++ b/github-project-export/tests/test_export_example_live.py @@ -19,6 +19,8 @@ _FIXTURES = Path(__file__).resolve().parent / "fixtures" _FIXTURE_INPUT = _FIXTURES / "fixture_1_input.json" _FIXTURE_OUTPUT = _FIXTURES / "fixture_1_output.tsv" +_FIXTURE_2_INPUT = _FIXTURES / "fixture_2_input.json" +_FIXTURE_2_OUTPUT = _FIXTURES / "fixture_2_output.tsv" def _subprocess_env() -> dict[str, str]: @@ -90,3 +92,39 @@ def test_export_example_matches_golden() -> None: f"TSV mismatch (normalized by url column).\n" f"--- expected ---\n{expected}\n--- actual ---\n{actual}\n--- stderr ---\n{proc.stderr}" ) + + +@pytest.mark.integration +@pytest.mark.skipif( + _needs_github_token(), + reason="GITHUB_TOKEN not set (or GH_TOKEN); e.g. GITHUB_TOKEN=$(gh auth token) uv run pytest", +) +def test_export_or_query_matches_golden() -> None: + """OR-query fixture: two milestones unioned, deduplicated, sorted by url.""" + expected_raw = _FIXTURE_2_OUTPUT.read_text(encoding="utf-8") + expected = _normalize_tsv(expected_raw) + + proc = subprocess.run( + [ + sys.executable, + "-m", + "github_project_export.cli", + str(_FIXTURE_2_INPUT), + "--quiet", + ], + cwd=_PACKAGE_ROOT, + capture_output=True, + text=True, + encoding="utf-8", + env=_subprocess_env(), + check=False, + ) + + assert proc.returncode == 0, ( + f"CLI failed (exit {proc.returncode})\nstderr:\n{proc.stderr}\nstdout:\n{proc.stdout}" + ) + actual = _normalize_tsv(proc.stdout) + assert actual == expected, ( + f"TSV mismatch (normalized by url column).\n" + f"--- expected ---\n{expected}\n--- actual ---\n{actual}\n--- stderr ---\n{proc.stderr}" + ) diff --git a/github-projects-client/github_projects_client/__init__.py b/github-projects-client/github_projects_client/__init__.py index 866a84e..9159791 100644 --- a/github-projects-client/github_projects_client/__init__.py +++ b/github-projects-client/github_projects_client/__init__.py @@ -4,6 +4,7 @@ from .fields import list_field_options from .items import list_items, list_fields, get_item from .mutations import set_field_value, set_field_value_bulk +from .query import expand_or_query from .views import resolve_view_url __all__ = [ @@ -16,5 +17,6 @@ "get_item", "set_field_value", "set_field_value_bulk", + "expand_or_query", "resolve_view_url", ] diff --git a/github-projects-client/github_projects_client/items.py b/github-projects-client/github_projects_client/items.py index ed09c86..d621d75 100644 --- a/github-projects-client/github_projects_client/items.py +++ b/github-projects-client/github_projects_client/items.py @@ -7,6 +7,7 @@ import requests from .api import fetch_items_rest, list_field_ids_by_name +from .query import expand_or_query DEFAULT_FIELDS = [ @@ -192,12 +193,18 @@ def list_items( fields: Optional[List[str]] = None, per_page: int = 50, cursor: Optional[str] = None, + max_or_items: int = 1000, ) -> Dict[str, Any]: """List project items with optional filter query. Uses cursor-based pagination: each call fetches one page from the REST API. Pass the returned ``next_cursor`` back to get the next page. + OR queries (e.g. ``(branch1) OR (branch2)``) fetch all matching items in a + single request and do not support cursor-based pagination. Passing a + ``cursor`` with an OR query raises ``ValueError``. The ``max_or_items`` + parameter caps the total items returned by OR queries (default 1000). + Returns a dict with: "items": list of compact formatted dicts (~200-300 bytes each) "next_cursor": cursor string to fetch the next page, or None @@ -224,17 +231,54 @@ def list_items( resolved_fields.append(f"{board_name} (id: {fid})") break - fetch_result = fetch_items_rest( - session, - org=org, - project_number=project_number, - query=query, - field_ids=field_ids if field_ids else None, - per_page=per_page, - max_pages=1, - cursor=cursor, - ) - raw_items = fetch_result["items"] + queries = expand_or_query(query) + + if len(queries) > 1 and cursor is not None: + raise ValueError( + "Cursor-based pagination is not supported for OR queries. " + "OR queries fetch all matching items in a single request." + ) + + if len(queries) == 1: + # Single query: standard cursor-based pagination (one page) + fetch_result = fetch_items_rest( + session, + org=org, + project_number=project_number, + query=queries[0], + field_ids=field_ids if field_ids else None, + per_page=per_page, + max_pages=1, + cursor=cursor, + ) + raw_items = fetch_result["items"] + next_cursor = fetch_result["next_cursor"] + has_more = fetch_result["has_more"] + else: + # Multiple OR branches: fetch all pages for all queries, deduplicate. + # Safety cap to prevent runaway queries from overwhelming the server. + raw_items: List[Dict[str, Any]] = [] + seen_ids: set[str] = set() + for q in queries: + fetch_result = fetch_items_rest( + session, + org=org, + project_number=project_number, + query=q, + field_ids=field_ids if field_ids else None, + per_page=per_page, + max_pages=None, + ) + for item in fetch_result["items"]: + node_id = _extract_node_id(item) + if node_id not in seen_ids: + seen_ids.add(node_id) + raw_items.append(item) + if len(raw_items) >= max_or_items: + raw_items = raw_items[:max_or_items] + break + next_cursor = None + has_more = False items = [_format_item(item, fields) for item in raw_items] @@ -245,13 +289,13 @@ def list_items( "resolved_fields": resolved_fields, "per_page": per_page, "items_returned": len(items), - "has_more": fetch_result["has_more"], + "has_more": has_more, } return { "items": items, - "next_cursor": fetch_result["next_cursor"], - "has_more": fetch_result["has_more"], + "next_cursor": next_cursor, + "has_more": has_more, "debug": debug, } diff --git a/github-projects-client/github_projects_client/query.py b/github-projects-client/github_projects_client/query.py new file mode 100644 index 0000000..822083f --- /dev/null +++ b/github-projects-client/github_projects_client/query.py @@ -0,0 +1,152 @@ +"""OR-condition query expansion for GitHub Projects v2 filter syntax.""" + +from __future__ import annotations + + +def expand_or_query(query: str) -> list[str]: + """Expand a query with OR syntax into multiple individual queries. + + Terms before the first parenthesized group form a shared prefix that is + prepended to every OR branch. Each ``(...)`` group becomes a separate + query string. Groups are separated by the ``OR`` keyword (uppercase). + + Returns a single-element list when no OR syntax is present (passthrough). + + Raises ``ValueError`` for malformed expressions (unmatched parens, nested + parens, trailing terms, empty groups, ``OR`` without parens, ``OR`` inside + parens). + """ + groups: list[str] = [] + prefix_parts: list[str] = [] + current: list[str] = [] + in_quotes = False + paren_depth = 0 + found_parens = False + found_or = False + expect_group = False # True after OR — next token must be '(' + expect_or = False # True after ')' — next token must be OR or end + i = 0 + n = len(query) + + while i < n: + ch = query[i] + + # Toggle quote state + if ch == '"': + in_quotes = not in_quotes + current.append(ch) + i += 1 + continue + + if in_quotes: + current.append(ch) + i += 1 + continue + + # Open paren + if ch == "(": + if paren_depth == 1: + raise ValueError("Nested parentheses are not supported") + if expect_or: + raise ValueError( + "Expected OR between groups; consecutive groups require OR" + ) + paren_depth = 1 + found_parens = True + expect_group = False + # Everything collected before first group (and not after a group) is prefix + if not groups and not found_or: + token = "".join(current).strip() + if token: + prefix_parts.append(token) + current = [] + elif current: + # Text between ')' and '(' that isn't OR + token = "".join(current).strip() + if token: + raise ValueError( + "Filter terms after the last group are not allowed" + ) + current = [] + i += 1 + continue + + # Close paren + if ch == ")": + if paren_depth == 0: + raise ValueError("Unexpected closing parenthesis") + paren_depth = 0 + group_content = "".join(current).strip() + if not group_content: + raise ValueError("Empty group") + groups.append(group_content) + current = [] + expect_or = True + i += 1 + continue + + # Check for OR keyword (outside quotes, outside parens) + if paren_depth == 0 and ch in ("O", "o") and query[i : i + 2] == "OR": + # Ensure OR is whitespace-bounded + before_ok = (i == 0) or query[i - 1] in (" ", "\t") + after_ok = (i + 2 >= n) or query[i + 2] in (" ", "\t") + if before_ok and after_ok: + found_or = True + expect_or = False + expect_group = True + token = "".join(current).strip() + if token: + # Text before OR that's not in a group — means OR without parens + if not found_parens: + raise ValueError("OR requires parenthesized groups") + raise ValueError( + "Filter terms after the last group are not allowed" + ) + current = [] + i += 2 + continue + + # Check for OR inside parens + if paren_depth == 1 and ch in ("O", "o") and query[i : i + 2] == "OR": + before_ok = (i == 0) or query[i - 1] in (" ", "\t") + after_ok = (i + 2 >= n) or query[i + 2] in (" ", "\t") + if before_ok and after_ok: + raise ValueError("OR inside parentheses is not supported") + + current.append(ch) + i += 1 + + # Post-loop checks + if in_quotes: + raise ValueError("Unmatched quote") + + if paren_depth != 0: + raise ValueError("Unmatched opening parenthesis") + + if expect_group: + raise ValueError("OR must be followed by a parenthesized group") + + # No parens and no OR found — passthrough + if not found_parens and not found_or: + return [query] + + # OR found but no parens + if found_or and not found_parens: + raise ValueError("OR requires parenthesized groups") + + # Trailing terms after last group + trailing = "".join(current).strip() + if trailing: + raise ValueError("Filter terms after the last group are not allowed") + + prefix = " ".join(prefix_parts).strip() + + if not groups: + return [query] + + # Single group, no OR — strip parens and combine with prefix + if len(groups) == 1 and not found_or: + combined = f"{prefix} {groups[0]}".strip() + return [combined] + + return [f"{prefix} {g}".strip() for g in groups] diff --git a/github-projects-client/tests/test_integration.py b/github-projects-client/tests/test_integration.py index bebf2e4..c201bac 100644 --- a/github-projects-client/tests/test_integration.py +++ b/github-projects-client/tests/test_integration.py @@ -527,3 +527,53 @@ def test_invalid_ref_format(self, session: requests.Session): item_ref="not-a-valid-ref", ) assert details is None + + +# --------------------------------------------------------------------------- +# OR query support (via list_items) +# --------------------------------------------------------------------------- + + +class TestOrQuery: + """Tests for OR query expansion through list_items.""" + + def test_or_query_returns_union(self, session: requests.Session): + """OR query returns items from both branches.""" + result = list_items( + session, + org=FILOZ_ORG, + project_number=PROJECT_NUMBER, + query='is:issue (milestone:"M4.0: mainnet staged" no:assignee) OR (milestone:"M4.1: mainnet ready" has:assignee)', + fields=["Repository", "Id", "Title", "Milestone"], + per_page=50, + ) + assert len(result["items"]) > 0 + milestones = {item.get("Milestone", "") for item in result["items"]} + assert "M4.0: mainnet staged" in milestones, "Expected items from M4.0 branch" + assert "M4.1: mainnet ready" in milestones, "Expected items from M4.1 branch" + assert result["has_more"] is False + assert result["next_cursor"] is None + + def test_or_query_rejects_cursor(self, session: requests.Session): + """OR query raises ValueError when a cursor is provided.""" + with pytest.raises(ValueError, match="not supported for OR queries"): + list_items( + session, + org=FILOZ_ORG, + project_number=PROJECT_NUMBER, + query='is:issue (milestone:"M4.0: mainnet staged") OR (milestone:"M4.1: mainnet ready")', + cursor="some_cursor_value", + ) + + def test_or_query_no_duplicates(self, session: requests.Session): + """OR query returns no duplicate node IDs.""" + result = list_items( + session, + org=FILOZ_ORG, + project_number=PROJECT_NUMBER, + query='is:issue (milestone:"M4.0: mainnet staged" no:assignee) OR (milestone:"M4.1: mainnet ready" has:assignee)', + fields=["Repository", "Id", "Title"], + per_page=50, + ) + node_ids = [item["_node_id"] for item in result["items"]] + assert len(node_ids) == len(set(node_ids)), "Duplicate _node_id values found" diff --git a/github-projects-client/tests/test_query_unit.py b/github-projects-client/tests/test_query_unit.py new file mode 100644 index 0000000..8e443ec --- /dev/null +++ b/github-projects-client/tests/test_query_unit.py @@ -0,0 +1,178 @@ +"""Unit tests for expand_or_query() parser.""" + +from __future__ import annotations + +import pytest + +from github_projects_client.query import expand_or_query + + +# --------------------------------------------------------------------------- +# Passthrough (no OR, no parens) +# --------------------------------------------------------------------------- + + +class TestPassthrough: + def test_plain_query(self): + assert expand_or_query("is:issue") == ["is:issue"] + + def test_complex_plain_query(self): + assert expand_or_query('is:issue -status:"🎉 Done"') == [ + 'is:issue -status:"🎉 Done"' + ] + + def test_empty_string(self): + assert expand_or_query("") == [""] + + def test_whitespace_only(self): + assert expand_or_query(" ") == [" "] + + def test_or_inside_quotes_is_literal(self): + q = 'title:"this OR that"' + assert expand_or_query(q) == [q] + + def test_parens_inside_quotes_are_literal(self): + q = 'title:"(hello) world"' + assert expand_or_query(q) == [q] + + +# --------------------------------------------------------------------------- +# Simple OR with prefix +# --------------------------------------------------------------------------- + + +class TestSimpleOr: + def test_two_branches_with_prefix(self): + result = expand_or_query( + 'is:issue (milestone:"M4.2" -status:"🎉 Done") OR (-last-updated:7days)' + ) + assert result == [ + 'is:issue milestone:"M4.2" -status:"🎉 Done"', + "is:issue -last-updated:7days", + ] + + def test_two_branches_no_prefix(self): + result = expand_or_query( + '(is:issue milestone:"M4.2") OR (is:pr -last-updated:7days)' + ) + assert result == [ + 'is:issue milestone:"M4.2"', + "is:pr -last-updated:7days", + ] + + def test_three_branches(self): + result = expand_or_query("is:issue (status:A) OR (status:B) OR (status:C)") + assert result == [ + "is:issue status:A", + "is:issue status:B", + "is:issue status:C", + ] + + def test_single_group_no_or_strips_parens(self): + result = expand_or_query('is:issue (milestone:"M4.2")') + assert result == ['is:issue milestone:"M4.2"'] + + +# --------------------------------------------------------------------------- +# Multi-branch OR +# --------------------------------------------------------------------------- + + +class TestMultiBranch: + def test_multi_word_prefix(self): + result = expand_or_query( + 'is:issue -status:"🎉 Done" (milestone:"M4.1") OR (milestone:"M4.2")' + ) + assert result == [ + 'is:issue -status:"🎉 Done" milestone:"M4.1"', + 'is:issue -status:"🎉 Done" milestone:"M4.2"', + ] + + def test_quoted_values_inside_groups(self): + result = expand_or_query('(status:"🏗 In Progress") OR (status:"📋 Backlog")') + assert result == [ + 'status:"🏗 In Progress"', + 'status:"📋 Backlog"', + ] + + +# --------------------------------------------------------------------------- +# OR/parens inside quotes (treated as literal) +# --------------------------------------------------------------------------- + + +class TestQuotedLiterals: + def test_or_in_quoted_prefix(self): + q = 'title:"error OR warning" (status:A) OR (status:B)' + result = expand_or_query(q) + assert result == [ + 'title:"error OR warning" status:A', + 'title:"error OR warning" status:B', + ] + + def test_parens_in_quoted_group(self): + q = '(title:"(important)") OR (status:B)' + result = expand_or_query(q) + assert result == [ + 'title:"(important)"', + "status:B", + ] + + +# --------------------------------------------------------------------------- +# Error conditions +# --------------------------------------------------------------------------- + + +class TestErrors: + def test_unmatched_open_paren(self): + with pytest.raises(ValueError, match="Unmatched opening parenthesis"): + expand_or_query("(a b") + + def test_unmatched_close_paren(self): + with pytest.raises(ValueError, match="Unexpected closing parenthesis"): + expand_or_query("a) OR (b)") + + def test_nested_parens(self): + with pytest.raises(ValueError, match="Nested parentheses"): + expand_or_query("((a)) OR (b)") + + def test_trailing_terms(self): + with pytest.raises(ValueError, match="Filter terms after the last group"): + expand_or_query("(a) OR (b) extra") + + def test_or_without_parens(self): + with pytest.raises(ValueError, match="OR requires parenthesized groups"): + expand_or_query("a OR b") + + def test_empty_group(self): + with pytest.raises(ValueError, match="Empty group"): + expand_or_query("(a) OR ()") + + def test_or_inside_parens(self): + with pytest.raises(ValueError, match="OR inside parentheses"): + expand_or_query("(a OR b)") + + def test_or_at_start_without_parens(self): + with pytest.raises( + ValueError, match="OR must be followed by a parenthesized group" + ): + expand_or_query("OR something") + + def test_or_at_end_without_parens(self): + with pytest.raises(ValueError, match="OR requires parenthesized groups"): + expand_or_query("something OR") + + def test_trailing_or_after_group(self): + with pytest.raises( + ValueError, match="OR must be followed by a parenthesized group" + ): + expand_or_query("(a) OR") + + def test_consecutive_groups_without_or(self): + with pytest.raises(ValueError, match="Expected OR between groups"): + expand_or_query("(a) (b)") + + def test_unmatched_quote(self): + with pytest.raises(ValueError, match="Unmatched quote"): + expand_or_query('is:issue title:"unclosed') diff --git a/specs/004-or-filter-syntax/checklists/requirements.md b/specs/004-or-filter-syntax/checklists/requirements.md new file mode 100644 index 0000000..97db30f --- /dev/null +++ b/specs/004-or-filter-syntax/checklists/requirements.md @@ -0,0 +1,35 @@ +# Specification Quality Checklist: OR-Condition Support for Search/Filter Syntax + +**Purpose**: Validate specification completeness and quality before proceeding to planning +**Created**: 2026-05-06 +**Feature**: [spec.md](../spec.md) + +## Content Quality + +- [x] No implementation details (languages, frameworks, APIs) +- [x] Focused on user value and business needs +- [x] Written for non-technical stakeholders +- [x] All mandatory sections completed + +## Requirement Completeness + +- [x] No [NEEDS CLARIFICATION] markers remain +- [x] Requirements are testable and unambiguous +- [x] Success criteria are measurable +- [x] Success criteria are technology-agnostic (no implementation details) +- [x] All acceptance scenarios are defined +- [x] Edge cases are identified +- [x] Scope is clearly bounded +- [x] Dependencies and assumptions identified + +## Feature Readiness + +- [x] All functional requirements have clear acceptance criteria +- [x] User scenarios cover primary flows +- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] No implementation details leak into specification + +## Notes + +- Updated 2026-05-06: Revised syntax from flat independent clauses to GitHub-style shared-prefix model (terms before first parenthesized group apply to all OR branches). +- All items pass validation. Spec is ready for `/speckit.clarify` or `/speckit.plan`. diff --git a/specs/004-or-filter-syntax/contracts/query-parser.md b/specs/004-or-filter-syntax/contracts/query-parser.md new file mode 100644 index 0000000..b96a9ab --- /dev/null +++ b/specs/004-or-filter-syntax/contracts/query-parser.md @@ -0,0 +1,66 @@ +# Contract: Query Parser (`expand_or_query`) + +## Function Signature + +``` +expand_or_query(query: str) -> list[str] +``` + +## Input + +A filter query string, as produced by config loading (`query` or joined `queryParts`). + +## Output + +A list of one or more expanded query strings, each ready to be sent as a `q` parameter to the GitHub Projects REST API. + +- **No OR present**: returns `[query]` (single-element list, passthrough) +- **OR present**: returns N strings, each being `"{shared_prefix} {group_contents}"` stripped of extra whitespace + +## Error Conditions + +Raises `ValueError` with a descriptive message for: + +| Condition | Example | Message pattern | +|-----------|---------|----------------| +| Unmatched `(` | `(a OR b` | "Unmatched opening parenthesis" | +| Unmatched `)` | `a) OR (b)` | "Unexpected closing parenthesis" | +| Nested parens | `((a)) OR (b)` | "Nested parentheses are not supported" | +| Trailing terms | `(a) OR (b) extra` | "Filter terms after the last group are not allowed" | +| OR without parens | `a OR b` | "OR requires parenthesized groups" | +| Empty group | `(a) OR ()` | "Empty group" | +| OR inside parens | `(a OR b)` | "OR inside parentheses is not supported" | + +## Behavioral Contract + +1. **Backward compatible**: Any query without `OR` (outside quotes) and without parentheses returns unchanged as a single-element list. +2. **Quote-aware**: `OR` and `()` inside double-quoted strings are treated as literal text. +3. **Idempotent**: Calling the function on an already-expanded query (no OR, no parens) is a no-op. +4. **Pure function**: No side effects, no I/O, no state. + +## Usage by Consumers + +### `export_rows()` (github-project-export) +``` +queries = expand_or_query(query) +for q in queries: + items += fetch_project_v2_items_rest(session, ..., query=q, ...) +deduplicate(items, key=lambda item: item["id"]) +``` + +### `list_items()` (github-projects-client) +``` +queries = expand_or_query(query) +if len(queries) == 1: + # Current single-page behavior with cursor +else: + # Fetch all pages for all queries, deduplicate by _node_id +``` + +### `load_export_config()` (github-project-export) +``` +try: + expand_or_query(query) # Validate at config load time +except ValueError as e: + raise ConfigError(str(e)) +``` diff --git a/specs/004-or-filter-syntax/data-model.md b/specs/004-or-filter-syntax/data-model.md new file mode 100644 index 0000000..d0f0003 --- /dev/null +++ b/specs/004-or-filter-syntax/data-model.md @@ -0,0 +1,50 @@ +# Data Model: OR-Condition Support + +## Entities + +### ExpandedQuery + +Represents the result of parsing a query string that may contain OR syntax. + +**Fields**: +- `prefix: str` — shared filter terms before the first parenthesized group (may be empty) +- `branches: list[str]` — contents of each parenthesized group (1+ elements) + +**Derived**: +- `queries: list[str]` — each branch combined with the prefix: `[f"{prefix} {branch}".strip() for branch in branches]` + +**Notes**: This is a conceptual entity. The implementation uses `expand_or_query()` which returns `list[str]` directly (the `queries` list). There is no need for a class — the function encapsulates the parsing. + +### ProjectItem (existing, unchanged) + +Raw REST item dict from the GitHub Projects v2 API. + +**Deduplication key**: `item["id"]` (numeric REST ID, always present) + +**Relevant fields for OR support**: +- `id: int` — unique REST numeric ID for the project item +- `node_id: str` — GraphQL node ID (e.g., `PVTI_lADOBt3abc...`) +- `content: dict` — embedded issue or PR object +- `fields: list[dict]` — project field values + +### FormattedItem (existing, unchanged) + +Dict produced by `_format_item()` in `items.py`. + +**Deduplication key**: `item["_node_id"]` (always populated) + +## State Transitions + +None. The feature is stateless — it parses a query, executes API calls, and merges results. + +## Validation Rules + +| Rule | Where enforced | +|------|---------------| +| Parentheses must be balanced | `expand_or_query()` | +| No nested parentheses | `expand_or_query()` | +| No trailing terms after last `)` | `expand_or_query()` | +| Each group must be non-empty | `expand_or_query()` | +| OR requires parenthesized groups | `expand_or_query()` | +| OR inside quotes is literal | `expand_or_query()` | +| Query string must be non-empty | `config_schema._require_non_empty_query()` (existing) | diff --git a/specs/004-or-filter-syntax/plan.md b/specs/004-or-filter-syntax/plan.md new file mode 100644 index 0000000..47a5482 --- /dev/null +++ b/specs/004-or-filter-syntax/plan.md @@ -0,0 +1,145 @@ +# Implementation Plan: OR-Condition Support for Search/Filter Syntax + +**Branch**: `004-or-filter-syntax` | **Date**: 2026-05-06 | **Spec**: [spec.md](spec.md) +**Input**: Feature specification from `/specs/004-or-filter-syntax/spec.md` + +## Summary + +Add GitHub-style OR syntax with shared-prefix semantics to the project filter system. Terms before the first parenthesized group are shared context prepended to every OR branch. Each branch becomes a separate REST API query; results are union-merged with deduplication by item ID. The parser lives in `github-projects-client` so both the export tool and MCP server benefit. + +## Technical Context + +**Language/Version**: Python >=3.13 +**Primary Dependencies**: `requests>=2.31`, `github-projects-client` (local editable) +**Storage**: N/A (stateless — reads from GitHub API, writes TSV) +**Testing**: pytest >=8.0 (export), >=9.0 (client); unit + integration with `@pytest.mark.integration` +**Target Platform**: CLI (macOS/Linux) +**Project Type**: CLI tool + library +**Performance Goals**: OR queries complete within the same time as running N sequential queries +**Constraints**: Backward compatibility with all existing configs +**Scale/Scope**: Typically 2-5 OR branches, hundreds of items per branch + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +Constitution is a blank template — no project-specific gates defined. Pass by default. + +## Project Structure + +### Documentation (this feature) + +```text +specs/004-or-filter-syntax/ +├── spec.md # Feature specification +├── plan.md # This file +├── research.md # Phase 0 output +├── data-model.md # Phase 1 output +├── quickstart.md # Phase 1 output +├── contracts/ # Phase 1 output +│ └── query-parser.md # Parser contract +└── checklists/ + └── requirements.md # Spec quality checklist +``` + +### Source Code (repository root) + +```text +github-projects-client/ +├── github_projects_client/ +│ ├── __init__.py # MODIFY: export expand_or_query +│ ├── api.py # NO CHANGE +│ ├── items.py # MODIFY: OR-aware list_items() +│ └── query.py # NEW: expand_or_query() parser +└── tests/ + ├── test_query_unit.py # NEW: parser unit tests + └── test_integration.py # MODIFY: add OR integration tests + +github-project-export/ +├── github_project_export/ +│ ├── config_schema.py # MODIFY: validate OR at config load +│ └── rest_export.py # MODIFY: multi-query + dedup in export_rows() +└── tests/ + ├── test_export_example_live.py # MODIFY: add OR golden file test + └── fixtures/ + ├── fixture_2_input.json # NEW: OR query input + └── fixture_2_output.tsv # NEW: OR query expected output +``` + +**Structure Decision**: Changes span two existing packages. No new packages or directories beyond `contracts/` in the spec folder. The parser is a single new module in the client library. + +## Implementation Phases + +### Phase 1: Parser (`github-projects-client/github_projects_client/query.py`) + +Pure function, no dependencies beyond stdlib. + +**Function**: `expand_or_query(query: str) -> list[str]` + +**Algorithm**: +1. Walk query char-by-char tracking `in_quotes` (toggled by `"`) and `paren_depth` (0 or 1). +2. Collect tokens outside parentheses as **shared prefix** (everything before first `(`). +3. Collect content inside `(...)` groups. +4. Between groups, require `OR` keyword (whitespace-separated). +5. If no `OR` and no parens found, return `[query]` (passthrough). +6. If single parens with no `OR`, return `[prefix + group_content]` (strip parens). +7. For N groups, return N strings: `[f"{prefix} {group}".strip() for group in groups]`. + +**Validation (raise `ValueError`)**: +- Unmatched parentheses +- Nested parentheses (depth > 1) +- Trailing terms after last `)` +- `OR` without parenthesized groups +- Empty group +- `OR` inside parentheses + +**Unit tests** (`tests/test_query_unit.py`): ~16 test cases covering passthrough, simple OR, multi-branch, quoted OR/parens, all error conditions. + +### Phase 2: Client integration (`github-projects-client`) + +**`items.py` — modify `list_items()`**: +- Import `expand_or_query` from `.query` +- At function entry, call `expand_or_query(query)` +- If single query: current behavior (one page, cursor-based) +- If multiple queries: for each expanded query, call `fetch_items_rest()` with `max_pages=None`, collect all raw items, deduplicate by `_node_id`, format, return with `has_more=False` + +**`__init__.py`**: Add `expand_or_query` to imports and `__all__`. + +### Phase 3: Export integration (`github-project-export`) + +**`rest_export.py` — modify `export_rows()`**: +- Import `expand_or_query` from `github_projects_client` +- After `build_columns()`, call `expand_or_query(query)` +- If single query: current behavior +- If multiple: loop `fetch_project_v2_items_rest()` per query, collect items, deduplicate by raw `"id"` field, then map to rows + +**`config_schema.py` — modify `_require_non_empty_query()`**: +- After assembling query string, call `expand_or_query()` wrapped in try/except +- Convert `ValueError` to `ConfigError` for early validation at config load time + +### Phase 4: Tests + +**Parser unit tests** (test_query_unit.py): see Phase 1. + +**Integration tests** (test_integration.py): Add `TestOrQuery` class: +- `test_or_query_returns_union`: OR query returns items from both branches +- `test_or_query_no_duplicates`: verify no duplicate `_node_id` values + +**Golden file test** (test_export_example_live.py): +- New fixture `fixture_2_input.json` with OR query +- New expected output `fixture_2_output.tsv` +- New test function comparing actual vs expected + +## Deduplication Strategy + +- **Raw items** (in `export_rows`): deduplicate by `item["id"]` (numeric REST ID, always present) +- **Formatted items** (in `list_items`): deduplicate by `item["_node_id"]` (always populated by `_format_item()`) +- First occurrence wins (preserves ordering from first branch that returned the item) + +## Verification + +1. Run parser unit tests: `cd github-projects-client && uv run pytest tests/test_query_unit.py -v` +2. Run client integration tests: `GITHUB_TOKEN=$(gh auth token) uv run pytest tests/test_integration.py -v` +3. Run export integration test: `cd github-project-export && GITHUB_TOKEN=$(gh auth token) uv run pytest -v` +4. Manual test with real OR config: create a config with `is:issue (milestone:"M4.2: mainnet GA" -status:"🎉 Done") OR (-last-updated:7days)` and verify output includes items from both branches +5. Backward compatibility: run existing fixture_1 test and verify identical output diff --git a/specs/004-or-filter-syntax/quickstart.md b/specs/004-or-filter-syntax/quickstart.md new file mode 100644 index 0000000..03ed465 --- /dev/null +++ b/specs/004-or-filter-syntax/quickstart.md @@ -0,0 +1,65 @@ +# Quickstart: OR-Condition Support + +## Using OR in the GitHub Project Exporter + +### 1. Create a config file with OR syntax + +```json +{ + "projectUrl": "https://github.com/orgs/FilOzone/projects/14", + "query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)", + "fields": ["Repository", "Id", "Title", "Status", "Milestone", "Assignees", "url"] +} +``` + +This retrieves: +- All unassigned issues in milestone M4.0, **plus** +- All assigned issues in milestone M4.1 + +The `is:issue` prefix applies to both branches. Each branch has different conditions that can't be expressed in a single query. + +### 2. Run the export + +```bash +cd github-project-export +GITHUB_TOKEN=$(gh auth token) uv run github-project-export my-config.json +``` + +### 3. Syntax reference + +``` +shared-prefix (branch-1) OR (branch-2) OR (branch-3) +``` + +- **Shared prefix**: terms before the first `(` — applied to every branch +- **Branches**: each `(...)` group contains branch-specific terms +- **OR**: uppercase, separates groups +- **No parens needed** when there's no OR (backward compatible) + +### More examples + +**Done issues plus recently updated (common real-world use case):** +```json +{ + "query": "is:issue (status:\"🎉 Done\") OR (-last-updated:7days)" +} +``` + +**Fully independent clauses (no shared prefix):** +```json +{ + "query": "(is:issue milestone:\"M4.1: mainnet ready\") OR (is:pr -last-updated:7days)" +} +``` + +**Using queryParts:** +```json +{ + "queryParts": [ + "is:issue", + "(milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)" + ] +} +``` + +The parts are joined first (`is:issue (milestone:...) OR (milestone:...)`), then OR parsing applies. diff --git a/specs/004-or-filter-syntax/research.md b/specs/004-or-filter-syntax/research.md new file mode 100644 index 0000000..3fef0b8 --- /dev/null +++ b/specs/004-or-filter-syntax/research.md @@ -0,0 +1,46 @@ +# Research: OR-Condition Support + +## R1: Syntax Model Selection + +**Decision**: GitHub-style shared-prefix with parenthesized OR branches. + +**Rationale**: Matches the model GitHub Issues search shipped in 2025 (`is:issue state:open (type:Bug OR type:Epic)`). Also aligns with Lucene and Jira JQL conventions where parentheses control operator precedence and terms outside groups apply to all branches. Users familiar with any of these systems will find the syntax intuitive. + +**Alternatives considered**: +- **Flat independent clauses** (`(full-clause-1) OR (full-clause-2)`): Simpler to parse but forces users to repeat shared context in every branch. Rejected because the user explicitly wanted shared prefix behavior. +- **Full boolean algebra** (nested AND/OR/NOT with arbitrary depth): More powerful but far more complex to implement. The union-of-queries model only needs one level of OR. Rejected as YAGNI — the use cases involve 2-5 independent branches with shared context. + +## R2: Parser Location + +**Decision**: New module `github_projects_client/query.py` in the shared client library. + +**Rationale**: The MCP server (`filozzy-mcp/server.py`) calls `list_items()` from the client library. Putting the parser in the client means the MCP server gets OR support automatically when `list_items()` is updated. The parser is a pure function (no deps), so it adds zero weight. + +**Alternatives considered**: +- **In github-project-export only**: Would require duplicating the logic later for the MCP server. Rejected. +- **In a new shared package**: Overkill for a single pure function. Rejected. + +## R3: Deduplication Key + +**Decision**: Use `item["id"]` (REST numeric ID) for raw items, `item["_node_id"]` for formatted items. + +**Rationale**: Every REST item has a unique numeric `id`. Formatted items always have `_node_id` populated by `_format_item()`. Both are stable identifiers for a project item. + +**Alternatives considered**: +- **Dedup by content URL** (`html_url`): Draft items may not have content URLs. Rejected. +- **Dedup by title**: Not unique. Rejected. + +## R4: Pagination Behavior with OR + +**Decision**: When OR is active in `list_items()`, fetch all pages for all branches, return complete set with `has_more=False`. + +**Rationale**: There is no meaningful single cursor for a multi-query operation. The project board is bounded (~hundreds of items per query), so fetching all pages is acceptable. The export tool already fetches all pages. + +**Alternatives considered**: +- **Compound cursor** (track position in each branch): Complex to implement, serialize, and resume. Rejected as YAGNI — result sets are small enough. + +## R5: OR Keyword Case Sensitivity + +**Decision**: `OR` must be uppercase only. + +**Rationale**: Avoids ambiguity with the lowercase word "or" appearing in filter values (e.g., `title:"error or warning"`). Matches Lucene convention (boolean operators must be ALL CAPS). diff --git a/specs/004-or-filter-syntax/spec.md b/specs/004-or-filter-syntax/spec.md new file mode 100644 index 0000000..18def02 --- /dev/null +++ b/specs/004-or-filter-syntax/spec.md @@ -0,0 +1,152 @@ +# Feature Specification: OR-Condition Support for Search/Filter Syntax + +**Feature Branch**: `004-or-filter-syntax` +**Created**: 2026-05-06 +**Status**: Draft +**Input**: User description: "Add OR-condition support to the project search/filter syntax, matching GitHub Issues search conventions. Terms outside parentheses are shared context applied to every OR branch. Under the covers, each branch becomes a separate API query (with shared prefix prepended), and results are union-merged with deduplication." + +## Syntax Design + +The syntax follows the same model as [GitHub Issues search](https://github.blog/developer-skills/application-development/github-issues-search-now-supports-nested-queries-and-boolean-operators-heres-how-we-rebuilt-it/): terms outside parenthesized groups are **shared context** that applies to every OR branch. + +### Examples + +**Basic OR with shared prefix:** +``` +is:issue (milestone:"M4.2: mainnet GA" -status:"🎉 Done") OR (-last-updated:7days) +``` +Expands to two queries: +1. `is:issue milestone:"M4.2: mainnet GA" -status:"🎉 Done"` +2. `is:issue -last-updated:7days` + +**Multiple OR branches:** +``` +is:issue -status:"🎉 Done" (milestone:"M4.2: mainnet GA") OR (milestone:"M4.1: mainnet ready") OR (-last-updated:7days) +``` +Expands to three queries: +1. `is:issue -status:"🎉 Done" milestone:"M4.2: mainnet GA"` +2. `is:issue -status:"🎉 Done" milestone:"M4.1: mainnet ready"` +3. `is:issue -status:"🎉 Done" -last-updated:7days` + +**No shared prefix (fully independent clauses):** +``` +(is:issue milestone:"M4.2: mainnet GA") OR (is:pr -last-updated:7days) +``` +Expands to two queries: +1. `is:issue milestone:"M4.2: mainnet GA"` +2. `is:pr -last-updated:7days` + +**No OR (backward compatible):** +``` +is:issue milestone:"M4.2: mainnet GA" -status:"🎉 Done" +``` +Works exactly as today — single query, no parsing changes. + +### Parsing Rules + +1. **Shared prefix**: Any filter terms appearing *before* the first parenthesized group are shared context, prepended to every OR branch. +2. **OR keyword**: Case-sensitive, uppercase only. Separates parenthesized groups. +3. **Parentheses**: Required when using OR — they delimit each branch's unique terms. Parentheses are stripped before sending to the API. +4. **Quoted values**: `OR` inside quoted values (e.g., `title:"OR gate design"`) is literal text, not an operator. +5. **No nesting**: Parentheses do not nest. Only one level of grouping is supported (sufficient for the union-of-queries model). +6. **No trailing terms**: Filter terms MUST NOT appear after the last parenthesized group. All shared context goes before the first group. + +## User Scenarios & Testing *(mandatory)* + +### User Story 1 - OR Query with Shared Prefix (Priority: P1) + +A project manager wants to export all issues from milestone M4.2 that aren't done, plus all issues updated within the last 7 days. The `is:issue` constraint applies to both branches. Today this requires two separate export runs and manual deduplication. With OR support, a single configuration file produces one unified, deduplicated result set. + +**Why this priority**: This is the core value proposition — combining disjoint filter criteria with shared context in a single query. + +**Independent Test**: Create a config file with `is:issue (milestone:"M4.2: mainnet GA" -status:"🎉 Done") OR (-last-updated:7days)`, run the exporter, and verify the output contains issues matching either branch (but no duplicates), and that all results are issues (not PRs). + +**Acceptance Scenarios**: + +1. **Given** a config with shared prefix `is:issue` and two OR branches, **When** the exporter runs, **Then** the shared prefix is applied to both branches, and the output contains the union of both result sets with no duplicates. +2. **Given** a config with three OR branches and a shared prefix, **When** the exporter runs, **Then** each branch inherits the shared prefix, and results are the union of all three queries. +3. **Given** an OR query where an item matches multiple branches, **When** the exporter runs, **Then** that item appears exactly once in the output. + +--- + +### User Story 2 - Backward-Compatible Single Query (Priority: P1) + +A user has existing config files with no OR conditions and no parentheses. These must continue to work identically — no changes to behavior, output, or error messages. + +**Why this priority**: Breaking existing configs would be unacceptable. This is a hard constraint, not a nice-to-have. + +**Independent Test**: Run all existing config examples and integration tests; output must be identical before and after the change. + +**Acceptance Scenarios**: + +1. **Given** an existing config file with a single `query` string (no OR, no parentheses), **When** the exporter runs, **Then** output is byte-for-byte identical to the output before this feature was added. +2. **Given** an existing config file using `queryParts` (no OR), **When** the exporter runs, **Then** output is identical to the output before this feature was added. + +--- + +### User Story 3 - Clear Error Messages for Malformed OR Queries (Priority: P2) + +A user writes a config with a malformed OR expression. The system provides a clear, actionable error message. + +**Why this priority**: Good error messages prevent frustration, but only matter once the feature itself works. + +**Independent Test**: Create config files with various malformed OR expressions and verify each produces a specific, helpful error message. + +**Acceptance Scenarios**: + +1. **Given** a query with `OR` but no parenthesized groups, **When** validation runs, **Then** a clear error explains that OR requires parenthesized groups. +2. **Given** a query with an empty group (e.g., `() OR (status:"Done")`), **When** validation runs, **Then** a clear error identifies the empty group. +3. **Given** a query with unmatched parentheses, **When** validation runs, **Then** a clear error points to the unmatched parenthesis. +4. **Given** a query with filter terms after the last parenthesized group, **When** validation runs, **Then** a clear error explains that shared terms must appear before the first group. + +--- + +### Edge Cases + +- What happens when all OR branches return zero results? The exporter produces a header-only TSV (consistent with current zero-result behavior). +- What happens when the OR keyword appears inside a quoted value (e.g., `title:"OR gate design"`)? It is treated as literal text, not as a logical operator. +- What happens with very large union result sets? Deduplication handles thousands of items without noticeable performance degradation. +- What happens when `OR` is used in `queryParts`? The array is first joined into a single string (as today), then OR/parentheses parsing applies to that joined string. +- What happens with no shared prefix and fully independent groups? (e.g., `(is:issue ...) OR (is:pr ...)`) — each group becomes a standalone query with no prefix prepended. +- What happens when parentheses appear but there is no `OR`? A single parenthesized group with no OR is treated as a plain query (parentheses are stripped). + +## Requirements *(mandatory)* + +### Functional Requirements + +- **FR-001**: The system MUST support an `OR` keyword (case-sensitive, uppercase only) that separates parenthesized filter groups within a query string. +- **FR-002**: Filter terms appearing before the first parenthesized group MUST be treated as a shared prefix, prepended to every OR branch before execution. +- **FR-003**: Each OR branch (shared prefix + group contents) MUST be executed as a separate server-side query, and results MUST be combined into a single deduplicated output. +- **FR-004**: Parentheses MUST be required when using OR to delimit each branch's unique terms. Parentheses are stripped before the query is sent to the API. +- **FR-005**: Deduplication of results across branches MUST use the item's unique project-item identifier so that an item appearing in multiple branch results appears exactly once in the final output. +- **FR-006**: When no `OR` keyword is present, the query MUST behave identically to the current implementation (full backward compatibility). A query with parentheses but no `OR` is treated as a plain query with parentheses stripped. +- **FR-007**: The `OR` keyword inside quoted values (e.g., `status:"OR something"`) MUST be treated as literal text, not as a logical operator. +- **FR-008**: The system MUST validate that: (a) every parenthesized group is non-empty, (b) parentheses are balanced, (c) no filter terms appear after the last group, and produce clear error messages for violations. +- **FR-009**: The `query` field and `queryParts` field in the JSON config MUST both support OR syntax. For `queryParts`, the array is first joined into a single string (as today), then OR parsing applies. +- **FR-010**: The system MUST preserve the existing column ordering, TSV formatting, and output behavior (stdout vs. file) regardless of whether OR is used. +- **FR-011**: Parentheses MUST NOT nest. Only one level of grouping is supported. + +### Key Entities + +- **Shared Prefix**: Filter terms before the first parenthesized group, applied to every OR branch. +- **OR Branch**: The contents of one parenthesized group, combined with the shared prefix to form a complete query. +- **Union Result Set**: The deduplicated combination of items returned from all branch queries. + +## Success Criteria *(mandatory)* + +### Measurable Outcomes + +- **SC-001**: Users can retrieve items matching any of up to 5 independent filter branches (with shared context) in a single export run, with results appearing within the same time as running those queries sequentially. +- **SC-002**: 100% of existing config files produce identical output after the change (zero regressions). +- **SC-003**: Duplicate items across branches are eliminated — the final output contains each unique item exactly once. +- **SC-004**: Users who make syntax errors in OR queries receive an error message that identifies the problem within 1 read (no guesswork required). + +## Assumptions + +- The GitHub Projects v2 REST API `q` parameter does not natively support OR logic, so OR must be implemented by issuing multiple requests and merging results client-side. +- Each project item has a stable unique identifier suitable for deduplication. +- The number of OR branches in practice will be small (typically 2-5), so the linear increase in API calls is acceptable. +- Row ordering in the final output does not need to be deterministic beyond what the existing tool provides (sorted by URL column in tests). +- The initial implementation targets the github-project-export tool; the shared client library will be enhanced so other tools (e.g., MCP server) can adopt OR support later. +- The `OR` keyword is case-sensitive (uppercase only) to avoid ambiguity with filter values that might contain the lowercase word "or". +- The syntax intentionally matches [GitHub Issues search conventions](https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/filtering-and-searching-issues-and-pull-requests) for familiarity: shared terms outside parentheses, `OR` between groups, no nesting. diff --git a/specs/004-or-filter-syntax/tasks.md b/specs/004-or-filter-syntax/tasks.md new file mode 100644 index 0000000..f5806b4 --- /dev/null +++ b/specs/004-or-filter-syntax/tasks.md @@ -0,0 +1,180 @@ +# Tasks: OR-Condition Support for Search/Filter Syntax + +**Input**: Design documents from `/specs/004-or-filter-syntax/` +**Prerequisites**: plan.md, spec.md, research.md, data-model.md, contracts/query-parser.md + +**Tests**: Unit tests for the parser are included (pure function, easy to test). Integration tests included for verification. + +**Organization**: Tasks are grouped by user story to enable independent implementation and testing. + +## Format: `[ID] [P?] [Story] Description` + +- **[P]**: Can run in parallel (different files, no dependencies) +- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3) +- Include exact file paths in descriptions + +## Phase 1: Setup + +**Purpose**: No new project scaffolding needed — changes are to existing packages. This phase creates the new parser module and its tests. + +- [X] T001 Create `expand_or_query()` parser function in `github-projects-client/github_projects_client/query.py` per contract in `specs/004-or-filter-syntax/contracts/query-parser.md` +- [X] T002 [P] Create unit tests for `expand_or_query()` in `github-projects-client/tests/test_query_unit.py` covering: passthrough (no OR), simple OR with prefix, OR without prefix, multi-branch OR, OR/parens inside quotes, and all error conditions (unmatched parens, nested parens, trailing terms, empty group, OR without parens, OR inside parens) +- [X] T003 Export `expand_or_query` from `github-projects-client/github_projects_client/__init__.py` + +**Checkpoint**: Parser is complete, unit tests pass. Run: `cd github-projects-client && uv run pytest tests/test_query_unit.py -v` + +--- + +## Phase 2: Foundational (Blocking Prerequisites) + +**Purpose**: No additional foundational work needed. The parser (Phase 1) is the only prerequisite for user story implementation. + +**⚠️ CRITICAL**: Phase 1 must be complete before proceeding. + +--- + +## Phase 3: User Story 1 - OR Query with Shared Prefix (Priority: P1) 🎯 MVP + +**Goal**: Users can run OR queries with shared prefix in the github-project-export tool. Each OR branch becomes a separate API query, results are union-merged with deduplication. + +**Independent Test**: Create a config file with `is:issue (milestone:"M4.2: mainnet GA" -status:"🎉 Done") OR (-last-updated:7days)`, run the exporter, verify output contains items from both branches with no duplicates. + +### Implementation for User Story 1 + +- [X] T004 [US1] Modify `export_rows()` in `github-project-export/github_project_export/rest_export.py` to call `expand_or_query(query)`, loop `fetch_project_v2_items_rest()` per expanded query, and deduplicate results by `item["id"]` before mapping to rows +- [X] T005 [US1] Add OR syntax validation to `github-project-export/github_project_export/config_schema.py` — in `_require_non_empty_query()`, call `expand_or_query()` and catch `ValueError` as `ConfigError` +- [X] T006 [US1] Create integration test fixture `github-project-export/tests/fixtures/fixture_2_input.json` with an OR query against FilOzone project #14 +- [X] T007 [US1] Create expected output `github-project-export/tests/fixtures/fixture_2_output.tsv` by running the OR query manually and capturing results +- [X] T008 [US1] Add golden-file test function in `github-project-export/tests/test_export_example_live.py` for the OR-query fixture + +**Checkpoint**: Export tool handles OR queries. Run: `cd github-project-export && GITHUB_TOKEN=$(gh auth token) uv run pytest -v` + +--- + +## Phase 4: User Story 2 - Backward-Compatible Single Query (Priority: P1) + +**Goal**: All existing config files with no OR conditions produce identical output after the change. + +**Independent Test**: Run existing fixture_1 test and verify identical output. + +### Implementation for User Story 2 + +- [X] T009 [US2] Verify existing integration test passes unchanged — run `cd github-project-export && GITHUB_TOKEN=$(gh auth token) uv run pytest tests/test_export_example_live.py::test_export_example_matches_golden -v` and confirm byte-for-byte identical output +- [X] T010 [US2] Verify `github-projects-client` integration tests pass unchanged — run `cd github-projects-client && GITHUB_TOKEN=$(gh auth token) uv run pytest tests/test_integration.py -v` + +**Checkpoint**: Zero regressions confirmed. All existing tests pass identically. + +--- + +## Phase 5: User Story 3 - Clear Error Messages for Malformed OR Queries (Priority: P2) + +**Goal**: Malformed OR expressions produce clear, actionable error messages. + +**Independent Test**: Create config files with malformed OR expressions and verify each produces a specific error. + +### Implementation for User Story 3 + +- [X] T011 [US3] Add error-case unit tests in `github-projects-client/tests/test_query_unit.py` for: OR at start/end, empty group, unmatched parens, trailing terms after last group (if not already covered by T002) +- [X] T012 [US3] Verify error messages propagate correctly through config loading — add test in `github-project-export/tests/` that loads a config with malformed OR and asserts `ConfigError` is raised with helpful message + +**Checkpoint**: All malformed queries produce clear errors. Parser unit tests and config validation tests pass. + +--- + +## Phase 6: MCP Server Integration + +**Goal**: The MCP server (`filozzy-mcp`) gets OR support automatically through `list_items()` in the shared client. + +### Implementation + +- [X] T013 Modify `list_items()` in `github-projects-client/github_projects_client/items.py` to call `expand_or_query(query)` — single query: current behavior; multiple queries: fetch all pages for all branches, deduplicate by `_node_id`, return with `has_more=False` +- [X] T014 [P] Add integration tests in `github-projects-client/tests/test_integration.py` — new `TestOrQuery` class with tests for OR union results and deduplication + +**Checkpoint**: MCP server supports OR queries through `list_items()`. Run: `cd github-projects-client && GITHUB_TOKEN=$(gh auth token) uv run pytest tests/test_integration.py::TestOrQuery -v` + +--- + +## Phase 7: Polish & Cross-Cutting Concerns + +**Purpose**: Documentation and final validation. + +- [X] T015 [P] Update `github-project-export/README.md` with OR syntax documentation and examples +- [X] T016 [P] Add OR-query example config in `github-project-export/examples/export.example3.json` +- [X] T017 Run `specs/004-or-filter-syntax/quickstart.md` validation — manually execute each example and verify correct output +- [X] T018 Run full test suite across both packages to confirm no regressions + +--- + +## Dependencies & Execution Order + +### Phase Dependencies + +- **Phase 1 (Setup)**: No dependencies — start immediately +- **Phase 3 (US1 - OR Query)**: Depends on Phase 1 (parser must exist) +- **Phase 4 (US2 - Backward Compat)**: Depends on Phase 3 (changes must be in place to verify no regressions) +- **Phase 5 (US3 - Error Messages)**: Depends on Phase 1 (parser error handling) +- **Phase 6 (MCP Integration)**: Depends on Phase 1 (parser must exist) +- **Phase 7 (Polish)**: Depends on Phases 3-6 + +### User Story Dependencies + +- **US1 (OR Query)**: Depends on parser (Phase 1) only +- **US2 (Backward Compat)**: Depends on US1 being complete (need to verify no breakage) +- **US3 (Error Messages)**: Can start after Phase 1, independent of US1 + +### Parallel Opportunities + +- T001 and T002 can be developed in parallel (parser + tests) +- T004 and T005 can be developed in parallel (different files) +- T006 and T007 are sequential (need to run query to capture output) +- T011 and T012 can be developed in parallel +- T013 and T014 can be developed in parallel +- T015 and T016 can be developed in parallel +- **US3 and Phase 6 can run in parallel** after Phase 1 + +--- + +## Parallel Example: Phase 1 + +``` +# These can run in parallel: +Task T001: Create parser in github-projects-client/github_projects_client/query.py +Task T002: Create unit tests in github-projects-client/tests/test_query_unit.py +``` + +## Parallel Example: User Story 1 + +``` +# These can run in parallel: +Task T004: Modify export_rows() in rest_export.py +Task T005: Add validation in config_schema.py +``` + +--- + +## Implementation Strategy + +### MVP First (User Story 1 Only) + +1. Complete Phase 1: Parser + unit tests +2. Complete Phase 3: Export tool OR support +3. **STOP and VALIDATE**: Test with real OR config against FilOzone project #14 +4. Confirm backward compatibility (Phase 4) + +### Incremental Delivery + +1. Parser + tests → Foundation ready +2. Export tool OR support → MVP! Test independently +3. Error message polish → Better UX +4. MCP server integration → Broader adoption +5. Documentation → Ship-ready + +--- + +## Notes + +- [P] tasks = different files, no dependencies +- [Story] label maps task to specific user story +- Commit after each phase completion +- The golden file fixture (T006/T007) requires a live GitHub token to generate expected output +- US2 is primarily a verification phase, not new implementation