win-capture: say which path component failed the hook trust check, and act only on the directory

[summeroff]

What

Two changes to the graphics hook trust check, both prompted by the first field report of the updater half of #756 failing.

1. Every trust refusal now says which path component was refused and why.
2. An untrusted ancestor is reported and acted on nowhere. Only the object — the shared hook directory, the payload directory, the files — decides anything.

Pairs with streamlabs/a-files-updater#144, which makes the same two changes there. hook-dir-security.h and its copy in the updater are kept in step by hand.

Why

The report was from the updater, but the cause sits in the rules both sides share. hook_path_chain_is_trusted() answers for the object and every directory above it, up to and including the drive root, and returns one bool. On the reporting machine that bool was false, and the log could not say whether the hook directory, %ProgramData% or C:\ had been refused, or on what. Working it out took a machine-local reimplementation of the rules, and the answer is still inference: the two failing checks share exactly one object, C:\.

That is fix 1. Fix 2 is what the same investigation turned up here: dir_was_trusted at the quarantine, was_trusted for the files, and the final gate were all the whole chain, so an ancestor we can neither vouch for nor repair was driving decisions that are only about the directory.

What it does now

hook_object_trust() takes an optional reason buffer; hook_path_trust() fills a struct hook_trust with the object and the ancestors answered separately, the failing ancestor named, and a reason for each — the owning SID, the granting SID with the rights it holds and the full mask, a reparse point, or the error that stopped the descriptor being read.

Paths are converted with os_wcs_to_utf8_ptr() rather than logged through %ls. %ls in a narrow printf converts via the CRT locale and drops the rest of the line at the first character outside it, and a user profile name is exactly where these paths carry one — which would lose the diagnostic for the users hardest to reach remotely.

Behaviour, for an untrusted ancestor only:

	before	after
shared hook directory	quarantined and recreated	left alone
hooks in it	treated as untrusted, reinstalled	trust judged on their own ACLs
install-directory payload	not published	published
result	HOOK_SHARED_UNSAFE	proceeds; logged
implicit vulkan layer	withdrawn	left registered
get_hook_path()	silent fallback to the install directory	uses the shared hook; a fallback now says why

Nothing changes when the object is untrusted. Same quarantine, same descriptor, same HOOK_SHARED_UNSAFE, same layer withdrawal.

hook_path_chain_is_trusted() is gone rather than left unused. Folding the two answers back into a single bool is what produced the behaviour above, and leaving the predicate in a shared header is how it comes back.

The relaxation, stated plainly

The layer is the part worth arguing about, and the argument is what settled it.

disable_vulkan_layer() removes the HKCU entry unconditionally and the HKLM one only when elevated. The app normally is not elevated. So withdrawing over an ancestor removed the per-user entry — which grants no privilege a standard user does not already have, since they can register their own implicit layer in their own hive whenever they like — and left the machine-wide entry, which is the one that reaches elevated processes and the one a standard user cannot create. The cost and the benefit were the wrong way round for the common case.

Against that, the cost is real and permanent: vulkan capture gone for every OBS derived application on the machine, for a condition no elevated writer can repair. Game capture was never affected — get_hook_path() falls back to the hook in the install directory — so this was only ever vulkan titles, but it was all of them.

And a standard user who can rename a parent of %ProgramData% can rename a parent of %ProgramFiles% and replace the application that gets launched. Whatever we decline to do about the hook is not their limiting factor. The same reasoning is why the install-directory payload is held to its own directory now rather than to its whole path.

Testing

• win-capture x64 RelWithDebInfo: builds clean, 0 warnings with /WX. Confirmed TreatWarningAsError and WarningLevel in the generated project first, then forced a full recompile of the translation unit rather than relying on an up-to-date target.
• clang-format 19.1.5 clean on both files. CI pins 19.1.1 — same major, not the same binary.
• Equivalence, not assumed. The header was compiled at streamlabs and at this revision into two harnesses and their answers diffed across 13 paths: identical on all of them. That covers the separator handling win-capture: restrict the shared graphics hook directory to administrators #756 added — a fully forward-slash path, a mixed one, and a trailing separator all still resolve the same way — so the trust computation is unchanged and only the policy differs. The harness also asserts the split agrees with the old combined answer on every path.
• Every reason branch was exercised against live ACLs in the same harness: the granting-ACE branch on C:\ProgramData (grants S-1-5-32-545 write access 0x00000116 — BUILTIN\Users, which is why the ancestor mask is weaker than the object one), the owner branch, a reparse point via C:\Documents and Settings, a missing path, and a path with no drive letter. C:\work\temp's ancestor failure was correctly attributed to C:\work.
• Probing C:\ as an object reports grants S-1-5-11 write access 0x00000004 — Authenticated Users with FILE_APPEND_DATA, the entry the weaker ancestor mask exists to tolerate. That is the mechanism behind the original report, and it is one inherit-only flag away from failing every path on a machine.
• Not run end to end: reproducing the reported condition needs a machine whose drive root ACL is non-default.

🤖 Generated with Claude Code

[summeroff]

Superseded by #762, which has the same two commits on a fresh branch. The automated review never picked this one up.