-
Notifications
You must be signed in to change notification settings - Fork 257
EDU-4420: Adds audit-stray-pages and updates READMEs #3562
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
93d22bc
EDU-4420: Adds audit-stray-pages and updates READMEs
3191f33
EDU-4220: Angela's additions
0dcaeec
EDU-4420: Update site with UTILITIES.md, add auditing tool to bin, etc
bbc7217
Merge branch 'main' of https://github.com/temporalio/documentation in…
0f7a93e
EDU-4420: Restore commented out audit logging
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,78 @@ | ||
# Temporal Documentation Site Utilities | ||
|
||
This file documents utilities meant to support repository health. | ||
These utilities are not meant for use by contributors. | ||
|
||
## audit-stray-pages | ||
|
||
[**Docusaurus**](https://docusaurus.io) serves all files, whether they are mentioned in sidebars.js or not. | ||
[**Algolia**](https://algolia.com) will index these files so they show up in search, even though we intend to hide them. | ||
|
||
The `audit-stray-pages` utility audits unreferenced docs files so they can be evaluated and, if needed, hidden in Docusaurus. | ||
This script scans your Docusaurus project for `.md` and `.mdx` files in the top level `docs/` folder and reports the ones that are **not actively referenced** in your `sidebars.js` file. "Not actively referenced" includes: | ||
|
||
- Files that were commented out in sidebars.js | ||
- Files that are not included at all | ||
|
||
It helps you find: | ||
|
||
- Stray or deprecated documentation pages that aren’t in the site navigation | ||
- Files that are publicly accessible but unintentionally published | ||
- Gaps in `id:` usage where default IDs don’t match sidebar references | ||
|
||
You can hide Docusaurus files by setting them to drafts in the file metadata: | ||
|
||
``` | ||
draft: true | ||
``` | ||
|
||
Docusaurus will render drafts in preview mode but not production. | ||
|
||
**Please note**: | ||
|
||
- Some "deprecated" files are left intentionally visible outside the normal navigation system and are linked from other pages. | ||
They should not be set to draft as that breaks `yarn build` and will fail CI and Vercel deployment. | ||
- The sidebars.js file does not use `/path/to/file.mdx`. | ||
It uses `/path/to/folder/your-file-id`, which uses the `id: your-file-id` declared in your mdx YAML frontmatter. | ||
- Slugs are not equivalent to ids. | ||
Ids are used to stitch together the site. | ||
|
||
### What It Does | ||
|
||
1. **Parses `sidebars.js`** to extract all actively referenced doc IDs | ||
- Ignores lines that are commented out (`//` or `/* ... */`) | ||
2. **Extracts doc IDs** from sidebar references, including: | ||
- Simple string references: `"foo/bar"` | ||
- Object syntax: `{ type: 'doc', id: 'foo/bar' }` | ||
3. **Scans all `.md` and `.mdx` files** in the `docs/` directory: | ||
- First checks if the file's path-derived ID is in the sidebar | ||
- Then checks for a frontmatter `id:` override and searches that | ||
- If neither is found, the file is flagged as unreferenced | ||
|
||
### Usage | ||
|
||
Save the script as `audit-stray-pages` and make it executable: | ||
|
||
```bash | ||
chmod +x audit-stray-pages | ||
./audit-stray-pages | ||
``` | ||
|
||
### Work | ||
|
||
After running the script, review each unreferenced file carefully. | ||
|
||
- Some files may be intentionally excluded from the sidebar, such as deprecated pages or internal drafts. | ||
- Others may be unintentional omissions that should be added to sidebars.js. | ||
- Others are intentionally excluded but not referenced from anywhere on the site. | ||
This is deliberate in the case of `tctl` pages. | ||
This is accidental for other pages. | ||
|
||
For each flagged file, check whether it: | ||
|
||
- Should be deleted or archived | ||
- Should have a `draft: true` frontmatter added (Docusaurus v3+) | ||
- Needs a matching `id:` in the frontmatter | ||
- Belongs in `sidebars.js` but was accidentally left out or commented out | ||
|
||
This utility highlights potential issues—it’s up to you to decide what belongs in our published documentation. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,40 @@ | ||
#!/bin/bash | ||
|
||
SIDEBARS_FILE="sidebars.js" | ||
CLEANED_SIDEBAR="/tmp/cleaned_sidebars.txt" | ||
|
||
# 1. Preprocess sidebars.js | ||
# Remove // comments and /* block comments */ | ||
sed '/^\s*\/\//d' "$SIDEBARS_FILE" | sed '/\/\*/,/\*\//d' > "$CLEANED_SIDEBAR" | ||
|
||
# 2. Extract IDs from cleaned sidebars | ||
# IDs can appear like "foo/bar" or id: "foo/bar" | ||
grep -oE '"[^"]+"' "$CLEANED_SIDEBAR" | tr -d '"' | sort | uniq > /tmp/sidebar_ids.txt | ||
|
||
# 3. Now audit all docs | ||
find docs \( -name "*.mdx" -o -name "*.md" \) | while read -r filepath; do | ||
# Get default ID | ||
default_id=$(echo "$filepath" | sed -E 's|^docs/||;s|\.mdx$||;s|\.md$||') | ||
|
||
# Check if default_id is in the sidebar IDs | ||
if grep -qxF "$default_id" /tmp/sidebar_ids.txt; then | ||
continue | ||
fi | ||
|
||
# If not found, check for custom ID | ||
custom_id=$(grep -m 1 '^id:' "$filepath" | sed 's/id:[[:space:]]*//' | tr -d '\r') | ||
if [ -n "$custom_id" ]; then | ||
custom_id=$(echo "$custom_id" | xargs) | ||
if grep -qxF "$custom_id" /tmp/sidebar_ids.txt; then | ||
continue | ||
fi | ||
fi | ||
|
||
# Neither found — this is a potential stray file | ||
# Removing something linked but deprecated will break the build | ||
# Prefer to remove intentional items (commented out in sidebars.js) | ||
# Investigate items that may be unintentional. | ||
# Try to add slugs to all affected files that match the file names | ||
# and search for those | ||
echo "$filepath" | ||
done |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.