Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
__pycache__/
*.pyc
*.egg-info/
.venv/
venv/
dist/
.claude/settings.local.json
.mcp.json
.env*
.DS_Store
5 changes: 5 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,14 @@ When contributing to this repository:
## Active Technologies
- Python >=3.13 (constraint from all `pyproject.toml` files) + `requests>=2.31` (shared client), `mcp>=1.0` (MCP server only) (003-generalize-mcp-client)
- `action_log.jsonl` (append-only, MCP layer only) (003-generalize-mcp-client)
- Python >=3.13 + `requests>=2.31`, `github-projects-client` (local editable) (004-or-filter-syntax)
- N/A (stateless β€” reads from GitHub API, writes TSV) (004-or-filter-syntax)

---

*This document should be updated as the repository evolves and new conventions are established.*



## Recent Changes
- 004-or-filter-syntax: Added Python >=3.13 + `requests>=2.31`, `github-projects-client` (local editable)
70 changes: 68 additions & 2 deletions github-project-export/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,8 @@ Exit codes: `0` success (including zero matching items β†’ header-only TSV), `1`
| Key | Required | Description |
|-----|----------|-------------|
| `projectUrl` | Yes | Org project URL: `https://github.com/orgs/ORG/projects/N` |
| `query` | No* | Single project filter string (`q`). If both `query` and `queryParts` exist, **non-empty `query` wins**. |
| `queryParts` | No* | Array of strings; joined with spaces to form `q` when `query` is not used (or is empty). **Every element must be a string.** |
| `query` | No* | Single project filter string (`q`). Supports OR syntax (see below). If both `query` and `queryParts` exist, **non-empty `query` wins**. |
| `queryParts` | No* | Array of strings; joined with spaces to form `q` when `query` is not used (or is empty). OR syntax applies after joining. **Every element must be a string.** |
| `fields` | Yes | Non-empty array of column headers. Order = TSV column order. |
| `outputFile` | No | `null` or omit β†’ stdout; non-empty string β†’ file path. **`""` is invalid.** |

Expand Down Expand Up @@ -89,10 +89,76 @@ Values come from the linked **issue or pull request** (`content`), not from cust

You still get a **TSV header row** and **no data rows**.

### OR syntax

Combine multiple filter branches with `OR`. Terms before the first parenthesized group form a **shared prefix** applied to every branch. Each branch becomes a separate API query; results are union-merged with deduplication.

```
shared-prefix (branch-1) OR (branch-2) OR (branch-3)
```

- **Shared prefix**: terms before the first `(` β€” applied to every branch
- **Branches**: each `(...)` group contains branch-specific terms
- **OR**: uppercase keyword separating groups
- **Backward compatible**: queries without `OR` or parentheses work unchanged

**Example** β€” unassigned items from one milestone, assigned items from another (each branch has different conditions that can't be expressed in a single query):

```json
{
"query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)"
}
```

This expands to two queries:
1. `is:issue milestone:"M4.0: mainnet staged" no:assignee`
2. `is:issue milestone:"M4.1: mainnet ready" has:assignee`

Items appearing in both branches are deduplicated by ID.

**Real-world use case** β€” all done issues plus recently updated issues (not suitable for golden files since the board changes, but the most common use):

```json
{
"query": "is:issue (status:\"πŸŽ‰ Done\") OR (-last-updated:7days)"
}
```

#### Limitations

This is single-level OR expansion, not full boolean search. Each OR branch becomes a separate API query whose results are unioned. It does **not** support the nested boolean grouping that GitHub Issues search and Lucene/Elasticsearch offer.

**Not supported:**

```
# Nested parentheses
((milestone:"M4.0" OR milestone:"M4.1") AND status:"πŸŽ‰ Done")
β†’ Error: Nested parentheses are not supported

# Filter terms after the last group
(status:A) OR (status:B) is:issue
β†’ Error: Filter terms after the last group are not allowed

# OR without parenthesized groups
is:issue OR is:pr
β†’ Error: OR requires parenthesized groups

# AND keyword (use the shared prefix instead)
(milestone:"M4.0") AND (status:"πŸŽ‰ Done")
β†’ Not recognized β€” AND is not a keyword; write: status:"πŸŽ‰ Done" (milestone:"M4.0")

# OR inside a single group
(milestone:"M4.0" OR milestone:"M4.1")
β†’ Error: OR inside parentheses is not supported
```

The shared-prefix model covers the most common real-world cases (2–5 branches with shared context). For queries that need multi-dimensional boolean grouping, run separate exports.

### Examples

- [examples/export.example1.json](examples/export.example1.json) β€” narrow filter, matches the live integration fixture.
- [examples/export.example2.json](examples/export.example2.json) β€” broader columns (milestone, assignees, reviewers, etc.).
- [examples/export.example3.json](examples/export.example3.json) β€” OR query with different conditions per branch.

## Implementation notes

Expand Down
14 changes: 14 additions & 0 deletions github-project-export/examples/export.example3.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"projectUrl": "https://github.com/orgs/FilOzone/projects/14",
"query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)",
"fields": [
"Repository",
"Id",
"Title",
"Status",
"Milestone",
"Assignees",
"url"
],
"outputFile": null
}
10 changes: 10 additions & 0 deletions github-project-export/github_project_export/config_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@
from pathlib import Path
from typing import Any, List, Optional

from github_projects_client import expand_or_query

from github_project_export.board_url import parse_org_project_url


Expand Down Expand Up @@ -43,6 +45,10 @@ def _require_non_empty_query(data: dict[str, Any]) -> str:
q_chars = q.strip() if isinstance(q, str) else ""

if q_chars:
try:
expand_or_query(q_chars)
except ValueError as e:
raise ConfigError(str(e)) from e
return q_chars

if qp is not None and len(qp) > 0:
Expand All @@ -51,6 +57,10 @@ def _require_non_empty_query(data: dict[str, Any]) -> str:
raise ConfigError(
"'query' is empty and 'queryParts' produces an empty filter",
)
try:
expand_or_query(joined)
except ValueError as e:
raise ConfigError(str(e)) from e
return joined

raise ConfigError(
Expand Down
37 changes: 29 additions & 8 deletions github-project-export/github_project_export/rest_export.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
import requests

from github_projects_client import (
expand_or_query,
fetch_items_rest as fetch_project_v2_items_rest,
list_field_ids_by_name as list_project_v2_field_ids_by_name,
)
Expand Down Expand Up @@ -187,12 +188,32 @@ def export_rows(
)
columns, field_ids = build_columns(fields, board_map)

result = fetch_project_v2_items_rest(
session,
org=org,
project_number=project_number,
query=query,
field_ids=field_ids if field_ids else None,
)
queries = expand_or_query(query)

return [_item_to_row(it, columns) for it in result["items"]]
if len(queries) == 1:
result = fetch_project_v2_items_rest(
session,
org=org,
project_number=project_number,
query=queries[0],
field_ids=field_ids if field_ids else None,
)
raw_items = result["items"]
else:
raw_items: List[Dict[str, Any]] = []
seen_ids: set[int] = set()
for q in queries:
result = fetch_project_v2_items_rest(
session,
org=org,
project_number=project_number,
query=q,
field_ids=field_ids if field_ids else None,
)
for item in result["items"]:
item_id = item.get("id")
if item_id not in seen_ids:
seen_ids.add(item_id)
raw_items.append(item)

return [_item_to_row(it, columns) for it in raw_items]
5 changes: 3 additions & 2 deletions github-project-export/tests/fixtures/fixture_1_output.tsv
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
Repository Id Title Status url
dealbot 411 Investigate staging Dealbot startup failure πŸŽ‰ Done https://github.com/FilOzone/dealbot/issues/411
filecoin-services 411 Cleanup Calibration datasets to (temporarily) reduce chain bandwidth pressure before GA πŸŽ‰ Done https://github.com/FilOzone/filecoin-services/issues/411
synapse-sdk 411 fix: error outputs out of lotus are weird πŸŽ‰ Done https://github.com/FilOzone/synapse-sdk/pull/411
filecoin-services 411 Cleanup Calibration datasets to (temporarily) reduce chain bandwidth pressure before GA πŸŽ‰ Done https://github.com/FilOzone/filecoin-services/issues/411
dealbot 411 Investigate staging Dealbot startup failure πŸŽ‰ Done https://github.com/FilOzone/dealbot/issues/411
filecoin-pin 411 filecoin-pin-upload action failing due to pnpm πŸŽ‰ Done https://github.com/filecoin-project/filecoin-pin/issues/411
14 changes: 14 additions & 0 deletions github-project-export/tests/fixtures/fixture_2_input.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{
"projectUrl": "https://github.com/orgs/FilOzone/projects/14",
"query": "is:issue (milestone:\"M4.0: mainnet staged\" no:assignee) OR (milestone:\"M4.1: mainnet ready\" has:assignee)",
"fields": [
"Repository",
"Id",
"Title",
"Status",
"Milestone",
"Assignees",
"url"
],
"outputFile": null
}
Loading
Loading