Skip to content

improve behavior in case of contradictory data #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 41 commits into from
Apr 14, 2025

Conversation

ptahmose
Copy link
Contributor

@ptahmose ptahmose commented Apr 4, 2025

Description

This PR aims to define "expected, correct and canonical behavior" for cases where contradictory information is found in a CZI-document. The goal is to have defined and consistent behavior even in cases where a malformed CZI is processed.
According to this definition, libCZI is modified to conform to this specification.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

locally

Checklist:

  • I followed the Contributing Guidelines.
  • I did a self-review.
  • I commented my code, particularly in hard-to-understand areas.
  • I updated the documentation.
  • I updated the version of libCZI following Versioning of libCZI depending on the type of change
    • Bug fix -> PATCH
    • New feature -> MINOR
    • Breaking change -> MAJOR
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.

ptahmose and others added 7 commits March 26, 2025 16:53
- Added `sub_block_directory_info_policy_` to `CCZIReader` to control sub-block directory information prioritization.
- Updated `OpenOptions` with a new enumeration for sub-block directory policies: `SubBlockDirectoryPrecedence`, `SubBlockHeaderPrecedence`, and `IgnoreDiscrepancy`.
- Implemented discrepancy checks in `ReadSubBlock` to enforce consistency between sub-block directory and header information, throwing exceptions when necessary.
- Introduced a new error code, `SubBlockDirectoryToSubBlockHeaderMismatch`, in `LibCZICZIParseException` for discrepancy detection.
- Added a test case, `CheckThatSubBlockInfoFromSubBlockDirectoryIsAuthorative`, to validate the new functionality.
- Made minor improvements to `CMemOutputStream`, including removing `virtual` from the destructor and adding an override specifier to the `Write` method.
- Removed commented-out code in `CCZIReader::ReadSubBlock` to streamline property assignments from `subBlkData`.
- Added `CreateCziDocumentOneSubblock4x4Gray8` function in `test_reader.cpp` to facilitate the creation of a CZI document with a 4x4 grayscale subblock.
- Updated the test `CheckThatSubBlockInfoFromSubBlockDirectoryIsAuthorative` to utilize the new function for generating CZI documents, ensuring it verifies the expected content correctly.
Renamed the test case `CheckThatSubBlockInfoFromSubBlockDirectoryIsAuthorative` to `CheckThatSubBlockInfoFromSubBlockDirectoryIsAuthorativeNoException` to reflect its updated focus. Added a new test case `CheckThatExceptionIsThrownWhenEnabledIfSubBlockDirectoryAndSubblockHeaderDiffer` to verify exception handling for mismatches between sub-block directory and header. Updated the setting of `subBlockDirectoryInfoPolicy` to use `static_cast` for improved type safety. Additional assertions were included to validate the expected exception behavior.
Modified the `ReadSubBlock` method to improve discrepancy checks between sub-block directory and header information, making the directory information authoritative by default. Added a new enumeration `SubBlockDirectoryInfoPolicy` in `libCZI.h` to define handling policies for discrepancies, along with operator overloads for flag manipulation. Updated test cases in `test_reader.cpp` to reflect these changes, ensuring accurate retrieval of information and added a test to verify exception handling for discrepancies.
@ptahmose ptahmose requested a review from Copilot April 4, 2025 19:07
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

Src/libCZI/CZIReader.cpp:318

  • Consider using the overloaded operator& for SubBlockDirectoryInfoPolicy instead of casting to uint8_t. For example, rewrite the condition as 'if (!(this->sub_block_directory_info_policy_ & OpenOptions::SubBlockDirectoryInfoPolicy::IgnoreDiscrepancy))' for improved readability and consistency.
    if (!((uint8_t)this->sub_block_directory_info_policy_ & (uint8_t)OpenOptions::SubBlockDirectoryInfoPolicy::IgnoreDiscrepancy))

Copy link

codecov bot commented Apr 4, 2025

Codecov Report

Attention: Patch coverage is 71.69811% with 90 lines in your changes missing coverage. Please review.

Project coverage is 66.57%. Comparing base (11015ae) to head (a753e1f).
Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
Src/libCZI/decoder_zstd.cpp 61.49% 72 Missing ⚠️
Src/libCZI/decoder_wic.cpp 0.00% 11 Missing ⚠️
Src/libCZI/CreateBitmap.cpp 93.84% 4 Missing ⚠️
Src/libCZI/decoder.cpp 66.66% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #130      +/-   ##
==========================================
+ Coverage   65.88%   66.57%   +0.68%     
==========================================
  Files          86       89       +3     
  Lines       11004    11227     +223     
==========================================
+ Hits         7250     7474     +224     
+ Misses       3754     3753       -1     
Flag Coverage Δ
windows-latest 66.57% <71.69%> (+0.79%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ptahmose added 3 commits April 5, 2025 23:51
- Modified `CreateBitmapFromSubBlock_Uncompressed` to accept a new parameter for handling uncompressed data size mismatches.
- Updated `CreateBitmapFromSubBlock` to utilize the new `CreateBitmapOptions` structure, allowing for better control over bitmap creation options.
- Introduced `CreateBitmapOptions` struct in the `libCZI` namespace to manage various bitmap creation options.
- Refactored `CreateBitmap` method in `CCziSubBlock` to accept the new options parameter.
- Added test cases to ensure correct behavior when sub-block data is too short, verifying that bitmaps are filled with zeroes as expected.
- Improved code clarity and maintainability by removing commented-out code and streamlining logic.
Modified Decode method signatures to accept pointers for pixel type, width, and height, improving flexibility. Introduced a new parameter in CreateBitmapFromSubBlock_JpgXr to manage bitmap size mismatches. Updated comments and documentation for clarity. Adjusted test cases to align with new signatures and added tests for handling subblock size mismatches. Enhanced error handling and logging for better robustness.
- Removed explicit pixel type check in `CreateBitmapFromSubBlock_JpgXr`, allowing for more flexible bitmap decoding.
- Updated `CBitmapOperations::CopyWithOffsetInfo` to use the pixel type from the decoded bitmap.
- Cleaned up error handling in `CJxrLibDecoder::Decode` by removing commented-out code.
- Standardized variable names in `test_reader.cpp` for consistency.
- Replaced direct `Lock()` calls with `ScopedBitmapLockerSP` in tests for better resource management.
- Added new test cases for handling subblocks with mismatched pixel types.
- Made minor formatting adjustments for improved code readability.
@ptahmose ptahmose added the cla Contributor License Agreement sent to Admin label Apr 6, 2025
@ptahmose ptahmose requested a review from a team April 7, 2025 08:24
@ptahmose ptahmose requested a review from Copilot April 7, 2025 08:41
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

Src/libCZI/CZIReader.cpp:334

  • [nitpick] The use of the magic constant '1' for checking the flag value reduces clarity; consider replacing it with a well-named constant or an enum method to improve readability.
if (((uint8_t)this->sub_block_directory_info_policy_ & 1) == (uint8_t)OpenOptions::SubBlockDirectoryInfoPolicy::SubBlockDirectoryPrecedence)

ptahmose and others added 9 commits April 7, 2025 11:26
Updated logic to manage discrepancies between sub-block-directory and sub-block-header using configuration options. Enhanced checks to throw exceptions on mismatches in pixel type, compression, coordinates, and dimensions. Simplified precedence condition for sub-block-directory information using a bitmask for clarity.
- Introduced `decoder_zstd.h` for Zstd compressed data handling.
- Enhanced `CreateBitmapFromSubBlock_JpgXr` and `CreateBitmapFromSubBlock_ZStd0` with new parameters for better mismatch handling.
- Updated `Decode` methods in relevant decoders to accept additional arguments for customization.
- Added utility functions for token matching in `decoder_zstd.cpp`.
- Created `CreateBitmapOptions` structure to manage bitmap size discrepancy options.
- Updated tests in `test_reader.cpp` to validate new Zstd functionality.
- Made minor formatting and documentation improvements.
Updated `libCZI_Site.h` to add detailed documentation for the `Decode` method, clarifying parameter usage and introducing a convenience overload.

In `subblock_cache.h`, replaced existing atomic types with `std::atomic<std::uint64_t>` and `std::atomic<std::uint32_t>` for improved type safety and clarity.
Updated `CreateBitmapFromSubBlock_ZStd0` to use correct physical sizes. Introduced new functions in `decoder_zstd.cpp` for better handling of Zstd data size mismatches. Refactored `IsTokenMatch` to utilize the new decoding functions. Updated tests in `test_reader.cpp` to reflect changes in exception handling and added new tests for size mismatch scenarios.
- Updated `CreateBitmapFromSubBlock_ZStd0` and `ZStd1` to use references for `SubBlockInfo`, improving performance.
- Added a parameter for handling data size mismatches in `CreateBitmapFromSubBlock_ZStd1`.
- Modified `libCZI::CreateBitmapFromSubBlock` to accommodate the new parameter.
- Introduced new functions in `decoder_zstd.cpp` for size mismatch handling and HiLo byte unpacking with error checks.
- Added new test cases in `test_reader.cpp` for Zstd1 compressed data size scenarios, including HiLo byte packing.
- Renamed existing tests to better reflect changes in resolution protocol handling.
@ptahmose ptahmose requested a review from Copilot April 8, 2025 11:39
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 23 out of 23 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (3)

Src/libCZI/decoder_zstd.cpp:75

  • [nitpick] The function 'DecodeAndHiLoBytePackRequireCorrectSize' uses 'BytePack' in its name, while a related function uses 'ByteUnpack'. Consider renaming it for consistency if the operation performed is actually unpacking.
shared_ptr<libCZI::IBitmapData> DecodeAndHiLoBytePackRequireCorrectSize(const void* ptrData, size_t size, libCZI::PixelType pixelType, uint32_t width, uint32_t height)

Src/libCZI/CZIReader.cpp:317

  • The strict mismatch check between sub-block directory and header information may be too rigid. Verify that this condition fully reflects the intended resolution protocol and consider adding more granular logging to aid debugging in case of expected discrepancies.
if (entry.PixelType != subBlkData.pixelType || entry.Compression != subBlkData.compression || ... )

Src/libCZI/CreateBitmap.cpp:130

  • Ensure that the logic for handling insufficient uncompressed data correctly calculates the remaining size and zero-fills incomplete rows. A detailed review of the loop logic and the 'remaining_size' updates is recommended to avoid potential data corruption.
if (expected_size <= size) { ... } else { /* zero-fill logic */ }

Renamed `DecodeAndHiLoByteUnpackAndHandleSizeMismatch` to `DecodeAndHiLoBytePackAndHandleSizeMismatch` to reflect its new functionality. Commented out several unused processing functions in `decoder_zstd.cpp`. Updated pointer casting in `test_reader.cpp` for improved type safety and added new test cases to validate exception handling for Zstd1 compression edge cases.
@ptahmose ptahmose requested a review from Copilot April 8, 2025 23:53
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

Src/libCZI/CZIReader.cpp:317

  • [nitpick] Consider refactoring the bitwise flag comparison for clarity, such as by introducing a helper function or a boolean variable to improve readability.
if (static_cast<std::underlying_type<OpenOptions::SubBlockDirectoryInfoPolicy>::type>(this->sub_block_directory_info_policy_ & OpenOptions::SubBlockDirectoryInfoPolicy::IgnoreDiscrepancy) == 0)

ptahmose and others added 3 commits April 9, 2025 12:50
Updated the `libCZI` project version from `0.63.2` to `0.64.0` in `CMakeLists.txt` and added a corresponding entry in `version-history.markdown` for the new "Resolution Protocol for Ambiguous or Contradictory Information".

Introduced new overloads for the `Decode` method in `CJxrLibDecoder` and `CZstd1Decoder`, simplifying the interface by removing pointer requirements for pixel type, width, and height.

Updated tests in `test_JxrlibCodec.cpp`, `test_ZstdDecode.cpp`, and `test_writer.cpp` to utilize the new method signatures, ensuring they validate expected behavior while improving code readability and consistency across the decoding interface.
Updated variable names for consistency, changing `ptrData` and `pixelType` to `ptr_data` and `pixel_type`. Enhanced error messages for better clarity regarding zstd-compressed data size issues. Refactored code for improved readability and maintainability, utilizing `GetZstdContentSizeOrThrow` for size checks. Standardized handling of decompressed sizes to ensure consistent references, enhancing the overall robustness of the decoding functions.
Included `./libCZI/Doc/resolution_protocol.markdown` in the list of source files for Doxygen processing to enhance documentation coverage.
@ptahmose ptahmose marked this pull request as ready for review April 10, 2025 10:45
@ptahmose ptahmose requested a review from Copilot April 10, 2025 10:45
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 22 out of 24 changed files in this pull request and generated 1 comment.

Files not reviewed (2)
  • CMakeLists.txt: Language not supported
  • Src/Doxyfile: Language not supported
Comments suppressed due to low confidence (2)

Src/libCZI/CreateBitmap.cpp:140

  • [nitpick] Consider renaming 'remaining_size' to 'data_remaining' to more clearly indicate its purpose.
size_t remaining_size = size;

Src/libCZI/CZIReader.cpp:317

  • [nitpick] Consider extracting the bitmask check into a named variable or helper function to improve readability.
if (static_cast<std::underlying_type<OpenOptions::SubBlockDirectoryInfoPolicy>::type>(this->sub_block_directory_info_policy_ & OpenOptions::SubBlockDirectoryInfoPolicy::IgnoreDiscrepancy) == 0)

ptahmose and others added 2 commits April 10, 2025 12:48
Introduced a new function `DecompressAndThrowIfError` to encapsulate decompression and error handling. This change replaces inline decompression logic in multiple locations, enhancing code readability and maintainability. Additionally, updated comments and parameter descriptions to clarify expected behavior and requirements.
Copy link
Contributor

@DaveyJonesBitPail DaveyJonesBitPail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved w/ recommendations/questions

@ptahmose ptahmose merged commit e80a136 into ZEISS:main Apr 14, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla Contributor License Agreement sent to Admin
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants