Auto-publish PyDABs reference docs to gh-pages on release - #6729
Sankalp-Mittal merged 7 commits into
Conversation
Add an update-pydabs-docs workflow that rebuilds the PyDABs Python reference docs and publishes the HTML to the gh-pages branch on each release tag (and via workflow_dispatch), replacing the manual copy. Modeled on update-schema-docs.yml: same runner group, JFrog/Go/uv setup, and gh-pages worktree + github-actions[bot] commit + direct push pattern. Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: 4db1247
Top 6 slowest tests (at least 2 minutes):
|
Waiting for approvalBased on git history, these people are best suited to review:
Eligible reviewers: Suggestions based on git history. See OWNERS for ownership rules. |
…ocs' into sankalp-mittal/pydabs-docs-gitworkflow
Add -W to the pydabs-docs sphinx-build command so warnings (e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump) fail the build instead of passing silently. This makes both the existing python_docs CI job and the new gh-pages publish workflow strict, surfacing docs breakage at bump time. Fix the one pre-existing warning this exposes: the Resources docstring referenced add_job with :meth:, but add_job is a generated method Sphinx cannot resolve. Use an inline literal instead. Co-authored-by: Isaac <no-reply@databricks.com>
Refer to the generated per-resource adders as ``add_<resource>`` rather than singling out add_job, which was only an example among add_cluster, add_app, etc. Co-authored-by: Isaac <no-reply@databricks.com>
| - name: Setup JFrog | ||
| uses: ./.github/actions/setup-jfrog | ||
|
|
||
| - name: Setup Go |
There was a problem hiding this comment.
do we need go here? you can just call uv and then we also don't need jfrog
There was a problem hiding this comment.
We can run uv here but then this would be a secondary source of truth for the flags, the first one being in the Taskfile, over time I think that could cause divergence
| - docs/_output/** | ||
| cmds: | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going" | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going -W" |
There was a problem hiding this comment.
I think this will fail .github/workflows/python_push.yml on PRs and main (which is a good signal to gain) - could you please double check?
There was a problem hiding this comment.
Since this PR is based on the PR in which I updated the docs, so this passes on CI here
There was a problem hiding this comment.
I didn't mean it will fail main, I meant to please check that python_push does run on PRs and on main (so if e.g. SDK bump brings in a breaking change, we get a failed CI run).
Checked this run and yes it does run.
| on: | ||
| push: | ||
| tags: | ||
| - "v[0-9]+.[0-9]+.[0-9]+*" |
There was a problem hiding this comment.
what if this workflow fails, will it be retried? How will we know docs aren't updated?
There was a problem hiding this comment.
it won't be autoretried (can be done manually), if the docs aren't updated (push to gh-docs fails) this should fail, and if the docs itself on main are not updated (shouldn't happen as check on each PR) then the stale content is published. But this shouldn't happen as all PR merged would have to check for the updated docs
There was a problem hiding this comment.
workflows can fail for all sorts of reasons (timeouts, etc). this workflow is not self-healing though, only runs once per release and if it fails we don't know. maybe you can make this a step in the release runbook for on-call to check that docs got updated?
Align with the release-* workflow family that runs post-release, and update the internal name: to match. Co-authored-by: Isaac <no-reply@databricks.com>
PyDABs doc generation is a lightweight Sphinx build, so the -large protected runner is overkill. Drop to the standard protected runner (matching tagging.yml, which is also tag-triggered and pushes to the repo). Co-authored-by: Isaac <no-reply@databricks.com>
Full history is not needed: the build reads only the current main tree, and the release-tag lookup uses tag refs (fetch-tags), not file content at historical tags. Full history was inherited from update-schema-docs, whose since_version.go reads `git show <tag>:...` per release. Co-authored-by: Isaac <no-reply@databricks.com>
| - docs/_output/** | ||
| cmds: | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going" | ||
| - "uv run --python 3.12 sphinx-build docs docs/_output --show-traceback --nitpicky --fresh-env --keep-going -W" |
There was a problem hiding this comment.
I didn't mean it will fail main, I meant to please check that python_push does run on PRs and on main (so if e.g. SDK bump brings in a breaking change, we get a failed CI run).
Checked this run and yes it does run.
| on: | ||
| push: | ||
| tags: | ||
| - "v[0-9]+.[0-9]+.[0-9]+*" |
There was a problem hiding this comment.
workflows can fail for all sorts of reasons (timeouts, etc). this workflow is not self-healing though, only runs once per release and if it fails we don't know. maybe you can make this a step in the release runbook for on-call to check that docs got updated?
…s#6727) ## Stack - databricks#6727 — generate the PyDABs `.rst` doc pages from the generated modules (base) ← **this PR** - databricks#6729 — publish the built HTML to `gh-pages` + make the docs build strict ## Why The PyDABs Python reference docs (https://databricks.github.io/cli/python/) went stale. Codegen now emits **31** resource modules under `python/databricks/bundles/*/`, but the Sphinx `.rst` pages were hand-written and listed only 5 (`core`, `jobs`, `pipelines`, `schemas`, `volumes`). ## What Generate the per-resource `.rst` pages and the `index.rst` toctree inside `pydabs-codegen`: - New `python/codegen/codegen/generated_docs.py` — `write_docs()` scans the generated `databricks/bundles/*/` package dirs (decoupled from `RESOURCE_NAMESPACE`, robust to future renames), writes one page per resource, and regenerates the `index.rst` toctree. Called from `main.py` after the existing generation. - Stale generated pages are removed on each run so a dropped resource loses its page. - `core.rst` stays hand-written. The 4 previously hand-written resource pages (`jobs`, `pipelines`, `schemas`, `volumes`) regenerate **byte-identical** — no diff. `index.rst`'s prose header is preserved verbatim; only its toctree grows. - Acronym titles (`SQL Warehouses`, `MCP Services`) handled via a small override map. - The doc structure lives in `doc_page.rst.tmpl` and `doc_index.rst.tmpl` (loaded via `string.Template`, like the existing `*.py.tmpl` codegen templates) so it can be reviewed separately from the generation code. - `python/Taskfile.yml`: added the generated doc outputs to the task's `generates:` list. Also fold in a related fix surfaced in review: some upstream API descriptions aren't valid reStructuredText and break the Sphinx build (e.g. `sql.SpotInstancePolicy`, whose comment is a hard-wrapped ASCII grid table docutils rejects as malformed). - New burn-down override list `jsonschema_patch.override_descriptions` rewrites such descriptions into valid RST until the proto comment is fixed upstream (see `sqlgateway/scheduler/api/proto/endpoint_common.proto`). The override raises if it becomes a no-op, flagging entries to remove once upstream lands. - The strict `-W` docs gate that surfaces any *new* such breakage (at PR/SDK-bump time) lands in databricks#6729. CI's `generate-check` + `git diff --exit-code` now enforces the documented resource list can never drift from the generated modules. ## Testing - `./task pydabs-codegen` — regenerates code + 31 `.rst` pages + `index.rst`; the 4 existing pages show no diff; idempotent on re-run. - `./task pydabs-docs` — Sphinx build succeeds and now emits all 32 `databricks.bundles.*.html` pages; the `sql_warehouses` page renders a proper table instead of the malformed one. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Integration test reportCommit: c4b2348
173 interesting tests: 127 FAIL, 46 MISS
Top 50 slowest tests (at least 2 minutes):
|
## What The `databricks-bundles` Python reference landing page (https://databricks.github.io/cli/python/) linked "What is Python support for Declarative Automation Bundles? (TBD)" to `#` — a dead placeholder. Point it at the published docs page instead and drop the "(TBD)" marker. ```diff -See `What is Python support for Declarative Automation Bundles? (TBD) <#>`_. +See `Bundle configuration in Python <https://docs.databricks.com/aws/en/dev-tools/bundles/python>`_. ``` The target, [Bundle configuration in Python](https://docs.databricks.com/aws/en/dev-tools/bundles/python), is live and covers Python support for bundles. ## Testing Text-only change to a hand-written docs page. It's a well-formed reStructuredText external hyperlink (no cross-reference, no malformed RST), so the strict (`-W`) Sphinx build introduced in databricks#6729 is unaffected. This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
Stack
.rstdoc pages from the generated modules (base)gh-pages+ make the docs build strict ← this PRChanges
Two related pieces of the PyDABs docs pipeline:
1. Auto-publish the docs to
gh-pageson release. A newupdate-pydabs-docsGitHub Actions workflow rebuilds the PyDABs Python reference docs and publishes the HTML to thegh-pagesbranch on every release tag (and on demand viaworkflow_dispatch), replacing the manual copy that had left https://databricks.github.io/cli/python/ stale (5 pages vs. the 32 the build now emits). Modeled onupdate-schema-docs.yml:pushtagsv[0-9]+.[0-9]+.[0-9]+*+workflow_dispatch.contents: write,id-token: write(setup-jfrog GOPROXY exchange).main, set up JFrog / Go / uv, thengit worktree addorigin/gh-pages.go tool -modfile=tools/task/go.mod task pydabs-docs.gh-pagespython/directory wholesale frompython/docs/_output/, so a dropped resource loses its page; top-level.nojekyll,index.html, andexperimental/are untouched.github-actions[bot]and push directly togh-pages, no-op when there is no diff.2. Make the docs build strict (
-W). Add-Wto thepydabs-docssphinx-build command so warnings — e.g. malformed upstream descriptions or unresolved cross-references introduced by an SDK bump — fail the build instead of passing silently. This makes both the existingpython_docsCI job (every PR, including SDK bumps) and the new publish workflow strict, surfacing docs breakage at bump time. Fixes the one pre-existing warning this exposes: theResourcesdocstring referencedadd_jobwith:meth:, butadd_jobis a generated method Sphinx cannot resolve — changed to an inline literal.Stacked on top of #6727, so the strict build already includes its generated pages and the override for malformed upstream descriptions.
Tests
From repo root,
./task pydabs-codegenthen./task pydabs-docsboth exit 0 under-W(theadd_jobfix + #6727's override clear the only warnings). YAML validated locally.This pull request and its description were written by Isaac.