Refactor OSD to Support Custom Catalogs and Controlled Module Updates

[KatsuhiroWatanabe]

Summary (Revised on 2025-10-06)

This pull request consolidates previous commits and introduces a comprehensive revision to improve clarity and maintainability.
Rather than adding new features, it focuses on preserving the current functionality of OSD while enhancing its customizability for enterprise IT teams.

It introduces support for custom offline catalog files in OSD deployments, along with improvements to post-network initialization and module update control.

These changes provide greater flexibility for enterprise IT teams to tailor the OSD environment.
However, when automatic module updates are suppressed, IT teams must take responsibility for maintaining catalog files manually.

Note: A previously proposed enhancement to Save-WebFile.ps1 for HTTP header validation has been removed from this PR, as PR #316—which introduces a retry mechanism to improve download robustness—has already been merged into master. While the approaches differ, further refinement of the header validation logic may still be beneficial.
Just a quick note — unless I’m mistaken, the current implementation using $remote = Invoke-WebRequest -UseBasicParsing -Method Head -Uri $SourceUrl may not fully handle scenarios where $SourceUrl is invalid or unreachable (e.g., $remote -eq $null). Please disregard if this has already been addressed elsewhere.

---

Background

These patches are designed for scenarios involving Start-OSDCloudGUI.

The OSD module published to the PowerShell Gallery is automatically updated up to four times a day, including refreshed OS and DriverPack catalog files (JSON/XML) stored in the cache folder. Because these catalogs are bundled with the module, running Import-Module -Force by default updates both the PowerShell scripts and the catalog files.

This automation makes it difficult for enterprise IT teams to customize and test the OSD module, as any local modifications are overwritten by the latest version from the PowerShell Gallery.

To address this, the update introduces mechanisms to override default catalogs and suppress automatic module updates.

---

Key Enhancements

1. Offline Catalog Override

Affected files:

• Get-OSDCloudDriverPacks.ps1
• Get-OSDCloudOperatingSystems.ps1
• Get-OSDCloudOperatingSystemsIndexMap.ps1
• Get-OSDCloudOperatingSystemsIndexes.ps1

When the USB\OSDCloud\Catalogs folder is present, these patched scripts prioritize user-defined catalog files over the default cache.

---

2. Post-Network Script Execution & Suppression of Auto Module Update

Affected files:

• Initialize-OSDCloudStartnet.ps1
• OSDCloudTemplate.ps1

Enhancements:

• Suppress Automatic Module Updates
  Automatic updates can now be disabled by setting "OSDAutoUpdate": false in OSDCloud\Config\Initialize-OSDCloudStartnet.json. This allows IT teams to maintain control over catalog versions and avoid unintended overwrites.

• Execute Custom Script After Network Initialization
  A custom PowerShell script can be triggered after a network connection (Wi-Fi or Ethernet) is established.
  The script path is defined via a new template variable pointing to OSDCloud\Config\StartNet2.

These enhancements give enterprise IT teams greater control over deployment behavior and catalog management.

---

3. Non-Intrusive Design

All enhancements are opt-in. Default behavior remains unchanged unless the relevant folders or configuration files are explicitly provided, ensuring full backward compatibility.

---

Integration Notes

This PR includes cherry-picked changes from PR #288 for integration with existing startup logic.

---

Revision History

Date	Description
2025-07-21	Cherry-picked feature/InintStartNet2 (PR #288) for integration
2025-09-15	Revised with consolidated commits and full rewrite of PR description
2025-10-06	Removed changes to Save-WebFile.ps1 due to overlap with PR #316

[skyblaster]

This looks excellent, and since you're only excluding C: from the search path, it should work with network boot scenarios where no USB drive is present (ie. embedded in the boot.wim at X:\OSDCloud\Catalogs).

I would love to be able to define custom paths such as the following, and then leverage the "not yet implemented" cache host variable (see #272), which looks like something we could do with this patch:
"Url": "{HOSTNAME}/OSDCloud/OS/24H2-latest.wim"

[KatsuhiroWatanabe]

Thank you for the feedback!

Originally, since the current OSD reloads catalog files from the cache under the module path, we wanted an option to use only company-validated catalogs (Drivers and ESDs) stored in a different location, rather than relying solely on the default module path.

While the OSD module regularly updates cached catalog files via GitHub Actions—allowing us to always use the latest versions—there is currently no way to specify and use a particular version of a catalog file.

That’s why I proposed this pull request: to optionally allow importing catalog files from drives (excluding C:) under the \OSDCloud\Catalogs path.

As you pointed out, this also enables scenarios where the catalog is embedded in the boot.wim (e.g., at X:\OSDCloud\Catalogs) or stored on USB devices. The object list will be generated from the last successfully loaded catalog file, so you can take advantage of this behavior if it fits your use case.

Our ultimate goal is to host validated catalog files either on USB or online, and always use the latest version from those sources.

I’m also planning to propose another pull request to:
Add an option to disable automatic updates of the OSD module (to prevent overwriting custom code), and
Support optional PowerShell script execution after network connection (e.g., to download company-maintained catalogs from the internet).

We don’t necessarily expect this pull request to be merged as-is, since we are already operating with a patched custom version internally. However, we would greatly appreciate it if equivalent functionality could be considered for inclusion in the official OSD module.

[skyblaster]

Hi @KatsuhiroWatanabe, do you happen to have an example user defined OS catalog you could share?
Have you tested it with the new GUI in OSD.Workspace invoked by Start-OSDCloudWorkflow or only with Start-OSDCloudGUI?

EDIT: Sorry for the dumb question. I see this PR is geared towards OSDCloud v1.
The new functions appear to be Get-OSDCatalogOperatingSystems and Get-OSDCatalogDriverPacks

[KatsuhiroWatanabe]

Hi @skyblaster,

Thanks for your comment!

Yes, I can share an example of a user-defined OS catalog. It's a very simple one — just a modified version of the original JSON (cache\archive-cloudoperatingsystems\cache\archive-cloudoperatingsystems) with updated values for the OS download URL and SHA1 hash. Since it's quite straightforward, I didn’t include it here, but I’d be happy to upload it if needed.
(In our case, we had to patch the ja-jp.esd, so we tweaked the JSON and uploaded the patched ESD to our Azure Blob. At this moment, the issue is resolved by using Config/Scripts/Startup/Shutdown scripts.)

Also, the catalog should work with both ESD and WIM files. If the image contains only a single edition, OSD defaults to using Index=1, so there's no need to specify the index manually in that case.

As you mentioned, this PR is specifically for OSDCloud v1, and it's designed to work with Start-OSDCloudGUI.

Thanks for pointing out the new GUI in OSD.Workspace invoked via Start-OSDCloudWorkflow — this is new to me, and I’ll definitely look into it further.

Appreciate your feedback as always!

[KatsuhiroWatanabe]

Update: Integrated Enhancements from feature/InintStartNet2 (PR:#288)

This PR now includes additional enhancements originally proposed in feature/InintStartNet2 (PR:#288), cherry-picked for integration:

• Post-Network Script Execution: Enables execution of a custom script after network initialization.
• Optional Suppression of Module Auto-Update: Prevents automatic updates of OSD modules when explicitly disabled via configuration.

These additions complement the offline catalog support introduced in this PR, enabling more deterministic and customizable OSDCloud initialization workflows—especially in offline or controlled environments.

By integrating both, we provide a more complete solution for environments requiring strict control over OSD behavior.

Note: feature/InintStartNet2 remains unchanged and will be reviewed separately. This integration is intended to accelerate adoption of both features in scenarios where they are used together.

[KatsuhiroWatanabe]

Hi @OSDeploy,

Following up on this PR (originally submitted as #278, which was later replaced) and also #288, which builds on this one. I understand things can get busy, so I just wanted to check in and see if there's anything I can do to assist with the review process.

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable, using --force-with-lease to maintain a clean and consistent history. Please let me know if any further changes are needed.

Just to clarify — this PR does not alter existing behavior unless the new functionality is explicitly enabled through additional configuration. It should be safe to merge without impacting current users, and I hope it can be considered for inclusion.

Appreciate all your work on the project!

[KatsuhiroWatanabe]

Hi @OSDeploy,

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.

[KatsuhiroWatanabe]

Hi @OSDeploy,

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.

[KatsuhiroWatanabe]

Hi @OSDeploy,

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.

[KatsuhiroWatanabe]

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.

[KatsuhiroWatanabe]

Hi @OSDeploy,
To simplify the commit history and make the PR easier to review, I’ve squashed the commits into a single one and force-pushed the branch using --force-with-lease. The PR remains unchanged in terms of functionality.
In addition, I’ve completely revised the PR description to reflect the consolidated changes and clarify the purpose and scope of this update.
Please take a moment to review the updated description when convenient.

[KatsuhiroWatanabe]

I have reopened this pull request as I mistakenly closed PR #284 instead of PR #288.
While @OSDeploy has indicated that this feature may not be adopted, I’ve decided to keep this pull request open for potential future consideration.
This request is not intended to add a new feature, but rather to preserve the current functionality of OSD while improving its customizability for users.
I hope this proposal can be reconsidered as a way to offer more flexibility without disrupting the existing behavior.

[KatsuhiroWatanabe]

Hi @OSDeploy,
I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.

[KatsuhiroWatanabe]

As PR #316 introduces a similar fix and has been merged into master, I’ve decided to remove my changes to keep this PR mergeable. Nonetheless, I believe there’s still room for further enhancement in the HTTP header validation. Just a quick note — unless I’m mistaken, the current implementation using $remote = Invoke-WebRequest -UseBasicParsing -Method Head -Uri $SourceUrl may not handle scenarios where $SourceUrl is invalid or unreachable (e.g., $remote -eq $null). Please disregard if this is already addressed elsewhere.

[everydayintech]

Just a quick note — unless I’m mistaken, the current implementation using $remote = Invoke-WebRequest -UseBasicParsing -Method Head -Uri $SourceUrl may not handle scenarios where $SourceUrl is invalid or unreachable (e.g., $remote -eq $null). Please disregard if this is already addressed elsewhere.

if $SourceUrl is invalid or unreachable, curl.exe will create an empty file and the function will exit, as it did before #316
I agree that this is not optimal, will have a look at this later.

[KatsuhiroWatanabe]

Hi @everydayintech, thank you for your clarification regarding the behavior before and after #316.

You're absolutely right—your changes in #316 did not alter the core behavior of the function when $SourceUrl is invalid or unreachable, and I appreciate your effort to maintain consistency.

That said, I’d like to clarify a subtle but important detail. When curl.exe is invoked after a failed Invoke-WebRequest, and the URL points to a file that has been removed from the vendor’s server (e.g., due to outdated catalog information), the resulting file is not technically empty. In my testing, curl.exe downloads an HTML error page—typically a branded 404 response from the host.

For example, I ran the following command:

Save-WebFile -SourceUrl "https://download.lenovo.com/pccbbs/mobiles/tp_l13-gen-6_s2-gen-10_l13-2-in-1-gen-6_s2-2-in-1-gen-10_intel_w11_24h2_202508.old" -DestinationName notEmpty.exe -DestinationDirectory C:\Temp\

The result was a 1245-byte .exe file containing HTML content—not an empty file. So while the function exits as it did before #316, the fallback behavior may still produce a non-empty file that contains error markup. This distinction matters because downstream logic might interpret a non-zero file size (with null $remoteLength) as a successful download.

I believe this behavior predates #316 and wasn’t introduced by your commit. However, it might be worth documenting or validating to prevent unintended use of such error pages as binaries (e.g., a driver pack)—especially in automated workflows.

[KatsuhiroWatanabe]

I've been continuously rebasing this branch onto the latest master to keep it up to date and mergeable. Please let me know if any further changes are needed.