Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graphics-hook: Fix crash with Vulkan DirectDisplay #11583

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

mbechard
Copy link

@mbechard mbechard commented Dec 3, 2024

Vulkan Direct Display uses extensions:
VK_KHR_display/VK_EXT_direct_mode_display
This workflow allows creating swapchains that arn't active on the desktop, so they don't have HWNDs.
Avoid trying to create swap_data when a HWND can't be found.

Fixes #11581

Description

VK_KHR_display/VK_EXT_direct_mode_display extensions created swapchains without HWND handles.
Avoid trying to work with HWND when they can't be found.

Motivation and Context

The old behavior causes crashes when applications using Direct Display.

How Has This Been Tested?

The only thing I tested was that the crash doesn't occur. If avoiding filling the swap_data has
greater ramifications, then I'll need feedback from devs more familiar with OBS.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)

Checklist:

  • My code has been run through clang-format.
  • I have read the contributing document.
  • My code is not on the master branch.
  • The code has been tested.
  • All commit messages are properly formatted and commits squashed where appropriate.
  • I have included updates to all appropriate documentation.

Vulkan Direct Display uses extensions:
VK_KHR_display/VK_EXT_direct_mode_display
This workflow allows creating swapchains that arn't active on the
desktop, so they don't have HWNDs.
Avoid trying to create swap_data when a HWND can't be found.

Fixes obsproject#11581
@WizardCM WizardCM added Bug Fix Non-breaking change which fixes an issue Windows Affects Windows labels Dec 3, 2024
@jjulianoatnv
Copy link

This change makes sense. Not every VkSurfaceKHR/VkSwapchainKHR is associated with an hwnd. Assuming there is an associated hwnd results in undefined behavior (possibly application crash.)

The only caller of add_surf_data() is OBS_CreateWin32SurfaceKHR(). When a VkSurfaceKHR is created using a Vulkan command other than VkCreateWin32SurfaceKHR, then add_surf_data is not called for that surface. vkCreateDisplayPlaneSurfaceKHR() is one example of an alternate way to create a surface. There might be additional ways to create a surface.

The proposed change adds defense against both of the following fallacies:

  • It is a flaw to assume that all methods of creating a surface are wrapped by the win-capture plugin.
  • It is a flaw to assume that all methods of creating a surface associate the surface with an hwnd.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Fix Non-breaking change which fixes an issue Windows Affects Windows
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Crash in graphics-hook64.dll on call to vkCreateSwapchainKHR when using Vulkan Direct Display
4 participants