Skip to content

OGSMOD-6769: Run Vulkan unit tests after OpenGL for Viewport Toolbox #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Oct 7, 2025
Merged

OGSMOD-6769: Run Vulkan unit tests after OpenGL for Viewport Toolbox #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
2a11a88
OGSMOD-6769: Run Vulkan unit tests after OpenGL for Viewport Toolbox
sm-adsk Sep 25, 2025
a03f8ff
Refactor the macro HVT_TEST and simplify tests.
sm-adsk Sep 30, 2025
98bde86
Replace all HVT_TEST_DEFAULT_BACKEND's with HVT_TEST.
sm-adsk Oct 1, 2025
4e9a16e
Add 'const uint8_t pixelCountThreshold = 1' to TestContext::validateI…
sm-adsk Oct 1, 2025
d414f15
Replace inline with static for global variables.
sm-adsk Oct 1, 2025
2ec443a
Revert static to inline for some variables.
sm-adsk Oct 2, 2025
d02add7
Remove obsolete gParameterizedTests variable and its use.
sm-adsk Oct 2, 2025
0f86209
Add Doxygen comments.
sm-adsk Oct 3, 2025
f16df9c
Remove extra spaces.
sm-adsk Oct 3, 2025
5bc25e1
Merge branch 'main' into manars/OGSMOD-6769/parameterize-unit-tests
sm-adsk Oct 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 84 additions & 2 deletions test/RenderingFramework/TestFlags.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,89 @@
//
#pragma once

#include <gtest/gtest.h>
#include <string>

#define HVT_TEST(TestSuiteName, TestName) \
std::string ParamTestName##TestName(const testing::TestParamInfo<std::string>& info) \
{ \
return info.param; \
} \
class TestName : public ::testing::TestWithParam<std::string> \
{ \
public: \
void HVTTest##TestName( \
[[maybe_unused]] const std::string& computedImageName, \
[[maybe_unused]] const std::string& imageFile); \
Copy link
Contributor

@hodoulp hodoulp Oct 1, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Suggestion] We missed perhaps something to limit changes and/or build issues. The parameter names are those used everywhere by default, but in some case some utests customize their content for specific needs. If you rename computedImageName -> defaultComputedImageName & imageFile -> defaultImageFile it could perhaps ease the refactoring work.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we can do it in a future PR.

}; \
/* TODO: Enable "Vulkan" backend when Vulkan support is complete and stable. \
Currently, only "OpenGL" is enabled for testing. */ \
INSTANTIATE_TEST_SUITE_P(TestSuiteName, TestName, ::testing::Values(/*"Vulkan",*/ "OpenGL"), \
ParamTestName##TestName); \
TEST_P(TestName, TestName) \
{ \
TestHelpers::gRunVulkanTests = (GetParam() == "Vulkan"); \
TestHelpers::gTestNames = TestHelpers::getTestNames( \
::testing::UnitTest::GetInstance()->current_test_info()); \
const std::string imageFile = TestHelpers::gTestNames.suiteName + \
std::string("/") + TestHelpers::gTestNames.fixtureName; \
const std::string computedImageName = TestHelpers::appendParamToImageFile(imageFile); \
HVTTest##TestName(computedImageName, imageFile); \
} \
void TestName::HVTTest##TestName( \
[[maybe_unused]] const std::string& computedImageName, \
[[maybe_unused]] const std::string& imageFile)

namespace TestHelpers
{
inline bool gRunVulkanTests = false;
} // namespace TestHelpers
struct TestNames
{
/// @brief The name of the test suite extracted from the test information
std::string suiteName;
/// @brief The name of the test fixture extracted from the test suite name
std::string fixtureName;
/// @brief The parameter name extracted from the test name for parameterized tests
std::string paramName;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add Doxygen comments to explain these three data members.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

};

inline bool gRunVulkanTests = false;
inline TestNames gTestNames = TestNames {};

inline TestNames getTestNames(const ::testing::TestInfo* testInfo)
{
TestNames testNames;
if (testInfo)
{
std::string testSuiteName = testInfo->test_suite_name();
std::string testName = testInfo->name();

size_t pos = testSuiteName.find('/');
if (pos != std::string::npos)
{
testNames.suiteName = testSuiteName.substr(0, pos);
testNames.fixtureName = testSuiteName.substr(pos + 1);
}

pos = testName.find('/');
if (pos != std::string::npos)
{
testNames.paramName = testName.substr(pos + 1);
}
}
return testNames;
}

/// Gets the image file based on the test parameter.
inline std::string getComputedImagePath()
{
return gTestNames.paramName.empty() ? gTestNames.fixtureName
: (gTestNames.fixtureName + "_" + gTestNames.paramName);
}

/// Appends image file based on the test parameter.
inline std::string appendParamToImageFile(const std::string& fileName)
{
return gTestNames.paramName.empty() ? fileName : (fileName + "_" + gTestNames.paramName);
}

} // namespace TestHelpers
18 changes: 18 additions & 0 deletions test/RenderingFramework/TestHelpers.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,15 @@ bool HydraRendererContext::compareImages(
return compareImages(inFile, outFile, threshold, pixelCountThreshold);
}

bool HydraRendererContext::compareImage(const std::string& computedFilename,
const std::string& baselineFilename, const uint8_t threshold, const uint8_t pixelCountThreshold)
{
const auto baselinePath = getBaselineFolder();
const std::string baseline = getFilename(baselinePath, baselineFilename);
const std::string computed = getFilename(outFullpath, computedFilename + "_computed");
return compareImages(computed, baseline, threshold, pixelCountThreshold);
}

bool HydraRendererContext::compareOutputImages(const std::string& fileName1,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this method be removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is still used in other places.

const std::string& fileName2, const uint8_t threshold, const uint8_t pixelCountThreshold)
{
Expand Down Expand Up @@ -353,6 +362,15 @@ void TestContext::run(TestHelpers::TestStage& stage, hvt::Viewport* viewport, si
_backend->run(render, viewport->GetLastFramePass());
}

bool TestContext::validateImages(const std::string& computedImageName, const std::string& imageFile,
const uint8_t threshold, const uint8_t pixelCountThreshold)
{
if (!_backend->saveImage(computedImageName)) {
return false;
}
return _backend->compareImage(computedImageName, imageFile, threshold, pixelCountThreshold);
}

FramePassInstance FramePassInstance::CreateInstance(std::string const& rendererName,
pxr::UsdStageRefPtr& stage, std::shared_ptr<TestHelpers::HydraRendererContext>& backend,
std::string const& uid)
Expand Down
9 changes: 9 additions & 0 deletions test/RenderingFramework/TestHelpers.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,12 @@ class HydraRendererContext
virtual bool compareImages(const std::string& fileName, const uint8_t threshold = 1,
const uint8_t pixelCountThreshold = 0);

/// Compare image against stored "_computed" image and throws if a difference is found within
/// the threshold defined.
virtual bool compareImage(const std::string& computedFilename,
const std::string& baselineFilename, const uint8_t threshold = 1,
const uint8_t pixelCountThreshold = 1);

/// Compare two "_computed" images and throws if a difference is found within the thresholds
/// defined
virtual bool compareOutputImages(const std::string& fileName1, const std::string& fileName2,
Expand Down Expand Up @@ -227,6 +233,9 @@ class TestContext
// Render a viewport i.e., several frame passes.
void run(TestHelpers::TestStage& stage, hvt::Viewport* viewport, size_t frameCount);

bool validateImages(const std::string& computedImageName, const std::string& imageFile,
const uint8_t threshold = 1, const uint8_t pixelCountThreshold = 1);

public:
// The GPU backend used by the unit test.
std::shared_ptr<TestHelpers::HydraRendererContext> _backend;
Expand Down
2 changes: 1 addition & 1 deletion test/howTos/howTo01_CreateHgiImplementation.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@
//
// How to create an Hgi implementation?
//
TEST(howTo, createHgiImplementation)
HVT_TEST(howTo, CreateHgiImplementation)
{
pxr::HgiUniquePtr hgi;
pxr::HdDriver hgiDriver;
Expand Down
11 changes: 4 additions & 7 deletions test/howTos/howTo02_CreateOneFramePass.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,12 @@

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Already included in #include <RenderingFramework/TestContextCreator.h>

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hodoulp True, but isn't it good practice to include what is used in a cpp?
This makes the code of this file more robust to future changes in TestContextCreator.h, and it is also clearer, in my opinion.

That being said, this file will compile without the include, so you are right about that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In principle yes. In that specific case it adds useless changes everywhere.
Anyway, that's really minor.


//
// How to create one frame pass using Storm?
//
TEST(howTo, createOneFramePass)
HVT_TEST(howTo, createOneFramePass)
{
// Helper to create the Hgi implementation.

Expand Down Expand Up @@ -97,10 +99,5 @@ TEST(howTo, createOneFramePass)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
12 changes: 5 additions & 7 deletions test/howTos/howTo03_CreateTwoFramePasses.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

//
// How to create two frame passes?
//
Expand All @@ -35,9 +37,9 @@
// NOTE: It turns out "axisTripod.usda" has coplanar geometry and it can create random
// inconsistencies on all platforms. Refer to OGSMOD-6304.
#if (TARGET_OS_IPHONE == 1) || (defined(_WIN32) && defined(ENABLE_VULKAN)) || defined(__linux__) || !defined(ADSK_OPENUSD_PENDING)
TEST(howTo, DISABLED_createTwoFramePasses)
HVT_TEST(howTo, DISABLED_createTwoFramePasses)
#else
TEST(howTo, DISABLED_createTwoFramePasses)
HVT_TEST(howTo, DISABLED_createTwoFramePasses)
#endif
{
// Helper to create the Hgi implementation.
Expand Down Expand Up @@ -200,9 +202,5 @@ TEST(howTo, DISABLED_createTwoFramePasses)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());
ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile, 1));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
15 changes: 6 additions & 9 deletions test/howTos/howTo04_CreateACustomRenderTask.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,17 @@

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

//
// How to create a custom render task?
//
// TODO: The result image is not stable between runs on macOS, skip on that platform for now
// Disabled for Android due to baseline inconsistancy between runners. Refer to OGSMOD-8067
// Disabled for Android due to baseline inconsistency between runners. Refer to OGSMOD-8067
#if defined(__APPLE__) || defined(__ANDROID__)
TEST(howTo, DISABLED_createACustomRenderTask)
HVT_TEST(howTo, DISABLED_createACustomRenderTask)
#else
TEST(howTo, createACustomRenderTask)
HVT_TEST(howTo, createACustomRenderTask)
#endif
{
// Helper to create the Hgi implementation.
Expand Down Expand Up @@ -137,10 +139,5 @@ TEST(howTo, createACustomRenderTask)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
13 changes: 5 additions & 8 deletions test/howTos/howTo05_UseSSAORenderTask.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,15 +26,17 @@ PXR_NAMESPACE_USING_DIRECTIVE

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

//
// How to use the SSAO render task?
//
// FIXME: The result image is not stable between runs on macOS. Refer to OGSMOD-4820.
// Note: As Android is now built on macOS platform, the same challenge exists!
#if defined(__APPLE__) || defined(__ANDROID__)
TEST(howTo, DISABLED_useSSAORenderTask)
HVT_TEST(howTo, DISABLED_useSSAORenderTask)
#else
TEST(howTo, useSSAORenderTask)
HVT_TEST(howTo, useSSAORenderTask)
#endif
{
// Helper to create the Hgi implementation.
Expand Down Expand Up @@ -149,10 +151,5 @@ TEST(howTo, useSSAORenderTask)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
13 changes: 5 additions & 8 deletions test/howTos/howTo06_UseFXAARenderTask.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,14 +26,16 @@ PXR_NAMESPACE_USING_DIRECTIVE

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

//
// How to use the FXAA render task?
//
// FIXME: OGSMOD-7891 - Produce weird image & unstable images between runs on Android.
#if defined(__APPLE__) || defined(__ANDROID__)
TEST(howTo, DISABLED_useFXAARenderTask)
HVT_TEST(howTo, DISABLED_useFXAARenderTask)
#else
TEST(howTo, useFXAARenderTask)
HVT_TEST(howTo, useFXAARenderTask)
#endif
{
// Helper to create the Hgi implementation.
Expand Down Expand Up @@ -134,10 +136,5 @@ TEST(howTo, useFXAARenderTask)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
24 changes: 8 additions & 16 deletions test/howTos/howTo07_UseIncludeExclude.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ PXR_NAMESPACE_USING_DIRECTIVE

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

//
// Include or exclude geometry prims on the fly?
//
Expand Down Expand Up @@ -111,9 +113,9 @@ void CreateTest(const std::shared_ptr<TestHelpers::TestContext>& context,
// FIXME: It's sometime failed to render on iOS.Refer to OGSMOD-6933.
// Need to investigate if Android has similar issue too
#if defined(__ANDROID__) || (TARGET_OS_IPHONE == 1)
TEST(howTo, DISABLED_useCollectionToExclude)
HVT_TEST(howTo, DISABLED_useCollectionToExclude)
#else
TEST(howTo, useCollectionToExclude)
HVT_TEST(howTo, useCollectionToExclude)
#endif
{
// Helper to create the Hgi implementation.
Expand All @@ -134,20 +136,15 @@ TEST(howTo, useCollectionToExclude)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}

// FIXME: It's sometime failed to render on iOS.Refer to OGSMOD-6933.
// Need to investigate if Android has similar issue too
#if defined(__ANDROID__) || (TARGET_OS_IPHONE == 1)
TEST(howTo, DISABLED_useCollectionToInclude)
HVT_TEST(howTo, DISABLED_useCollectionToInclude)
#else
TEST(howTo, useCollectionToInclude)
HVT_TEST(howTo, useCollectionToInclude)
#endif
{
// Helper to create the Hgi implementation.
Expand All @@ -168,10 +165,5 @@ TEST(howTo, useCollectionToInclude)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
13 changes: 5 additions & 8 deletions test/howTos/howTo08_UseBoundingBoxSceneIndex.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,13 +31,15 @@ PXR_NAMESPACE_USING_DIRECTIVE

#include <gtest/gtest.h>

#include <RenderingFramework/TestFlags.h>

// FIXME: Android unit test framework does not report the error message, make it impossible to fix
// issues. Refer to OGSMOD-5546.
//
#if defined(__ANDROID__) || TARGET_OS_IPHONE == 1
TEST(howTo, DISABLED_useBoundingBoxSceneIndexFilter)
HVT_TEST(howTo, DISABLED_useBoundingBoxSceneIndexFilter)
#else
TEST(howTo, useBoundingBoxSceneIndexFilter)
HVT_TEST(howTo, useBoundingBoxSceneIndexFilter)
#endif
{
// This unit test demonstrates how to add a scene index filter to draw a bounding box using
Expand Down Expand Up @@ -112,10 +114,5 @@ TEST(howTo, useBoundingBoxSceneIndexFilter)

// Validates the rendering result.

const std::string imageFile = std::string(test_info_->test_suite_name()) + std::string("/") +
std::string(test_info_->name());

ASSERT_TRUE(context->_backend->saveImage(imageFile));

ASSERT_TRUE(context->_backend->compareImages(imageFile));
ASSERT_TRUE(context->validateImages(computedImageName, imageFile));
}
Loading