Feature request: HDR/color managed output

[mstoeckl]

Now that Wayland compositors are slowly moving toward supporting color management and HDR, having swaybg provide color-managed buffers to the compositor will soon start to become possible. Implementing color management would permit cooler wallpapers allow for better representation of a wider range of images, specifically those with saturated colors and bright spots.

Swaybg would also be a decent client to test color management protocols on, because it is relatively simple and directly uses Wayland protocols. Support for specific input/output features can be added piece by piece, without needing to wait for toolkit support.

The main challenge appears to be that, at the moment, image library support for color management is weak, and support for color management would need to be added format by format; or one would need to move away from gdk_pixbuf+Cairo; a few alternatives, none particularly great, are noted in #32. On the other hand, just adding support for PNG images would probably be good enough to start.

One route to incrementally add color management is to start by adding support for image formats providing their colorspace as CICP, and to test with the very restricted color management protocol currently being used by Kwin and VK_hdr_layer. I've made a test branch that does this, for PNG formats; see mstoeckl/frog-v1 .

The branch mstoeckl/frog-v1 is based on top of the current 10-bit output support PR.

It takes as input PNG files which have the (draft) cICP chunk set, and furthermore use one of the few primary/transfer combinations from frog-color-management-v1 (i.e, BT709 or BT2020 primaries, and sRGB, gamma2.2, PQ, or linear transfer function.) Such images can be created with ffmpeg from a video or from a .hdr file. For example, applying the following command to one of the files from https://hdri-haven.com/
ffmpeg -i input_file.hdr -vf "libplacebo=color_primaries=bt2020:color_trc=smpte2084" -pix_fmt rgba64be output_file.png
would produce an image with a cICP chunk, like the following :

Other notes:

• Computing eotf/transfer functions takes energy, and to minimize this when drawing backgrounds the compositor should ideally be working from a (linear space, f16) buffer with the transfer function already applied. However, there is also an energy cost from reading larger buffers. If compositors do not opt to cache transformed buffers, it might still be worth it for swaybg to do this part of the color transformation itself, and then submit buffers in the compositors preferred format (most likely linear/f16).

[mstoeckl]

The color_management_v1 protocol has now been released in wayland-protocols 1.41.

My initial test branch (only supports PNG images with CICP chunks whose primaries and transfer functions are in the named list supported by the compositor, and (edit: no longer!) assumes immediate image description processing) is at mstoeckl/png-hdr. I've tested it against current KWin. CICP chunks are supported by libpng (since 1.6.45, released 2025-01-07), and by the JPEG XL and AVIF formats.

In addition to the image processing issues noted above, adding full support for color_management_v1 will require significant main loop refactoring. When an output is dirty, instead of immediately loading, rendering, and committing an image, swaybg may need to construct an image description and wait for the wp_image_description_v1::ready event before calling wp_color_management_surface_v1::set_image_description() and comitting. (In practice, with parametric image descriptions, this event will be sent by the compositor immediately, so one doesn't need to wait. However, compositors are likely to handle ICC files asynchronously (for example, using a sandboxed process) so significant delays may be common for ICC image descriptions. A hypothetical create_immed function (like that of linux-dmabuf) for ICC image descriptions would not be a usable alternative unless swaybg were to perfectly validate all image ICC profiles before sending them.)

Also: doing color management with the background color may be complicated, both in how we specify out-of-sRGB-gamut colors and how to correctly blend non-sRGB images with alpha onto the background color. (swaybg is currently using linear blending on sRGB content, which is mildly imperfect but hard to notice under typical use; but this will break more strongly with e.g. sRGB background and PQ images. The problem could be deferred to be compositor using a subsurface, but that would be energy-inefficient. The best solution may be to blend and submit linear, f16 buffers, applying transfer functions inside swaybg.)

[emersion]

I think using libpng directly is reasonable (and we can always keep the current pipeline as a fallback).

I'm not sure ICC profile support is desirable, partly because the background image is not really used for color critical work, partly because Sway will not support the ICC profile part of the color-management protocol (at least initially, only Weston supports it, KWin also has no plans to support it). So I think blocking while the image description is ready is a good way forward if it allows us to avoid refactoring everything.

how we specify out-of-sRGB-gamut colors

Probably this can be left for later.

how to correctly blend non-sRGB images with alpha onto the background color

We could maybe convert the background color from sRGB to PQ before painting it onto the buffer. This would still blend electrical values instead of optical values, but at least we wouldn't mix sRGB and PQ anymore.

blend and submit linear, f16 buffers

This will allocate more memory and require more bandwidth to read from the image. I think I'd prefer swaybg to do more processing at startup rather than have a higher runtime cost.

We could probably do the EOTF + blend + inverse EOTF per-pixel and store the final result in a 10-bit buffer?