Multiple issues with JPEG textures

[lexaknyazev]

Follow-up to KhronosGroup/glTF#2471.

Some JPEG textures used in this repo violate the glTF 2.0 specification, namely they do not contain a JFIF marker and in some cases also contain Exif blocks. Additionally, some JPEG textures contain Adobe's color transform markers, embedded ICC profiles and/or thumbnails.

Why not having JFIF markers is a problem

The JPEG standard does not define any color model or channel semantics. An arbitrary JPEG file may contain up to 255 color channels for a single image. This is partially resolved by additional standards like ITU-T Rec T.871 (aka JFIF), ITU-T Rec T.872, and CIPA DC-008 (aka Exif).

Unfortunately, the amount of possible proprietary and standardized combinations of different markers is so large that it's not feasible for a glTF implementation to support arbitrary JPEG files. Although some non-standard marker configurations are supported by popular libraries and image-editing apps, it's not reasonable to expect glTF implementations to knowingly support them as well.

The most portable and safe option is to require JFIF markers to restrict the JPEG scope as much as possible.

JPEG files not having JFIF markers are already invalid per the glTF 2.0 spec and will be flagged as errors by the next validator version.

Why having Exif markers is a problem

The Exif standard is officially incompatible with JFIF because both specs require putting their respective markers immediately after SOI. Therefore, a valid JFIF file having an Exif block would be invalid as per the Exif specification. On top of that, there are thousands of various Exif fields that may or may not affect image interpretation. glTF implementations should not need to handle that.

Why having Adobe's color transform markers is a problem

Such markers are generally incompatible with JFIF and they are redundant in all existing assets anyway. Their intended purpose is to better support printing systems.

Why having embedded ICC profiles is a problem

glTF 2.0 explicitly defines that all color textures are treated as having ITU-R 709 color gamut with non-linear sRGB encoding. Any custom ICC profile must be ignored. Opening a JPEG image with a custom color profile in an image editor may yield different results than using the same image as a glTF texture.

Why having thumbnails is a problem

They require extra storage and increase transmission size while being inaccessible to glTF applications anyway.

[javagl]

Their intended purpose is to better support printing systems.

What if I want to print my glTF asset? 🥺 😉

Some of the details here are going so deep into the guts of certain specifications that I could hardly say more than what I already said in the other issue. And that comment already alludes to what may be the more pressing questions for the specific context of the glTF-Sample-Assets:

1. How to we find out which sample assets do have these "issues"?

I assume that the answer is that some 'dev'/'in-progress' version of the validator could do a batch run here, and will report most or all of these issues. (And you probably already did that, otherwise, you wouldn't have opened this issue to begin with)

2. More difficult: How to fix these assets?

While it may be technically possible to do some batch processing of these sample assets, this could raise some concerns. On this level, this is structurally similar to #114 and the discussion around KhronosGroup/glTF#2368 (comment) : If (nearly) every asset here is essentially "generated" (i.e. post-processed and fixed) by the same tool, then this may 1. involve changes that are not anticipated (like changes in buffer layouts based on defaults or preferences of that tool), and 2. limit the variability of the sample assets along these dimensions.

If there are not "many" assets that violate these constraints, the second point may not be a concern.

[lexaknyazev]

The most important action item is to fix textures that do not have JFIF markers to keep the repo error-free. Other issues would be considered as warnings for now. Only 11 assets (not counting GLB and compressed variants) are affected.

1. CesiumMan
   • glTF/CesiumMan_img0.jpg
2. ChairDamaskPurplegold
   • glTF/chair_damask_basecolor.jpg
   • glTF/chair_damask_roughmetal.jpg
   • glTF/chair_label.jpg
   • glTF/chair_occlusion.jpg
   • glTF/chair_wood_albedo.jpg
     • Note that this file has an embedded author information that does not match the asset's README
   • glTF/chair_wood_normal.jpg
3. DragonAttenuation
   • glTF/checkerboard.jpg
     • This texture should not use JPEG. It is poorly compressed and has blocking artifacts on square boundaries. An equivalent PNG with optimal encoding would be cleaner and about 60 times smaller.
4. EnvironmentTest
   • glTF/EnvironmentTest_images/roughness_metallic_0.jpg
   • glTF/EnvironmentTest_images/roughness_metallic_0.jpg
     • These two textures should not use JPEG. They are poorly compressed and have blocking artifacts on square boundaries. Equivalent PNGs with optimal encoding would be about 12 times smaller.
5. GlassBrokenWindow
   • glTF/WindowFrame_Occlusion.jpg
   • glTF/WindowClasp_Occlusion.jpg
   • glTF/WindowGlass_OcclusionRoughMetal.jpg
6. InterpolationTest
   • glTF/InterpolationTest_img0.jpg
     • This texture should not use JPEG. It is poorly compressed and has blocking artifacts. An equivalent PNG with optimal encoding would be cleaner and about 5 times smaller.
7. IridescenceAbalone
   • glTF/IridescenceAbalone_ORM.jpg
   • glTF/IridescenceAbalone_BaseColor.jpg
   • glTF/IridescenceAbalone_Iridescence.jpg
8. MandarinOrange
   • glTF/MandarinOrange_OcclusionRough.jpg
   • glTF/MandarinOrange_Basecolor.jpg
9. SheenCloth
   • glTF/SheenCloth_AO.jpg
10. TransmissionOrderTest
    • glTF/checkerboard.jpg
      • Same issue as in the other checkerboard.jpg above.
11. TransmissionTest
    • glTF/TransmissionTest_images/texture28577.jpg
      • Same issue as the checkerboard.jpg above.
    • glTF/TransmissionTest_images/texture14184.jpg
      • This texture should not use JPEG. An equivalent PNG with optimal encoding would be about 20 times smaller.
    • glTF/TransmissionTest_images/texture15366.jpg
      • This texture should not use JPEG. It is poorly compressed and has blocking artifacts. An equivalent PNG with optimal encoding would be about 7 times smaller.

[lexaknyazev]

Fixing those is quite easy with the exiftool.

Recursively listing all files without JFIF from the current directory

$ exiftool -FileName -r -ext jpg -if 'not $JfifVersion' .

Removing Adobe color transform markers (should be done first)

$ exiftool -overwrite_original -adobe= <FILE>

Adding a JFIF marker with resolution info taken from the existing Exif marker and removing everything else

$ exiftool -overwrite_original '-jfif:xresolution<xresolution' '-jfif:yresolution<yresolution' '-jfif:resolutionunit<resolutionunit' -all= <FILE>

[lexaknyazev]

The 11 assets using JPEG textures without JFIF markers have been fixed.

It's worth doing the following:

• Cleaning up the remaining texture files, i.e., removing Exif and ICC data, optimizing compression.
• Setting up the CI to flag image files that do not conform to the rules above (probably with some exceptions).
• Doing the same for image files used in documentation, i.e., screenshots.

[MarkCallow]

From where can exiftool be downloaded?

[javagl]

@MarkCallow That should be https://exiftool.org/ (I don't know that, but when there's https://en.wikipedia.org/wiki/ExifTool , it's a reasonable guess, at least...)