Crash in graphics-hook64.dll on call to vkCreateSwapchainKHR when using Vulkan Direct Display

[mbechard]

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

30.2.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/83BBFOiQVhsO906Y

OBS Studio Crash Log URL

No response

Expected Behavior

This isn't a crash in OBS Studio itself, rather the graphics-hook64.dll causing other applications to crash. OBS doesn't need to be running, just installed. Direct Display is a new Vulkan feature that allows an app to directly output to a monitor without interacting with the Windows desktop. I would not expect graphics-hook64.dll to cause other applications to crash, especially when OBS isn't even active.

Current Behavior

If OBS is installed on the machine, trying to use this feature crashes due to a bug in graphics-hook64.dll. Likely not handling this new extension properly.

Steps to Reproduce

1. You need Windows 11 Pro for Workstations or Windows 11 Enterprise to reproduce this. This is because the feature to 'remove a monitor from the desktop' is locked behind those versions.
2. Uninstall OBS, just so we can see the Direct Display working first.
3. Download and build Nvidia' sample project for Direct Display: https://github.com/nvpro-samples/vk_ddisplay
4. You can ignore the sections about mosaic and the config files. Just download the https://www.nvidia.com/en-us/drivers/ddisplay-utility/ and open an admin cmd prompt use:
5. configureDdisplay.exe status to see the displays on the machine
6. configureDdisplay.exe remove-from-desktop -i 1, for example, to remove monitor index 1 from the desktop.
7. Run the sample. You should see hairy torus object rotating on the monitor that isn't part of your Windows desktop anymore.
8. Shut down and install OBS, then try running again. It should crash.

Thanks for looking into this.

Anything else we should know?

No response