Consider adding a "LUT Generator" rendering path: NoSpatialAwareness

[gregcotten]

Summary

Users often want to create a LUT from their image pipeline.

DaVinci Resolve achieves this by piping a LUT image through the node graph. They've defined an extension OfxImageEffectPropNoSpatialAwareness that allows the plugin to specify whether or not it wants to participate in the LUT generation process. This approach is smart, but I think there could be better logic.

It seems like there should be three options for communicating this to the host:

1. The plugin will never affect spatial characteristics of the image, and a LUT can be generated at any time.
2. The plugin will always affect spatial characteristics of the image, and should not be used in the LUT generation process. (Probably should be the default if not specified?).
3. The plugin might currently affect spatial characteristics of the image, but when requested during a render (perhaps a property on the source clip) can temporarily disable those spatial characteristics to help generate a LUT. (imagine a plugin that applies film grain, blurs, any sort of geometric distortion, etc. but also non-spatial color characteristics)

It might also be useful to provide helper code for the host for generating a LUT image in-memory to pipe through their image pipeline.

Motivation

Users of hosts want the ability to generate LUTs. Hosts should have some built-in ability to do so. Plugins should be able to opt-in or opt-out.

Impact

Purely additive.

Documentation Impact

We'd need to add documentation, yes!

Stakeholders

Plugins and Hosts!

[barretpj]

Can we get docs on how Resolve's OfxImageEffectPropNoSpatialAwareness currently behaves?

I agree this is useful, and we should attempt to codify it in the standard.

[gregcotten]

/** @brief Indicates a plugin output does not depend on location or neighbours of a given pixel.
If the plugin is with no spatial awareness, it will be executed during LUT generation. Otherwise,
it will be bypassed during LUT generation.

    - Type - string X 1
    - Property Set - plugin descriptor (read/write)
    - Default - "false"
    - Valid Values - This must be one of
      - "false"  - the plugin is with spatial awareness, it will be bypassed during LUT generation
      - "true"   - the plugin is with no spatial awareness, it will be executed during LUT generation
 */
#define kOfxImageEffectPropNoSpatialAwareness "OfxImageEffectPropNoSpatialAwareness"

Here is the header information from Resolve's OpenFX "extras"

[barretpj]

The criteria would seem to be: one input clip, at the current time (not multi-input, not temporal), where each output pixel's colour depends purely on [the parameters and] the colour of the input pixel at the same location (not an area operation like a blur, nor a position-dependent operation like a vignette).

It might also be useful to provide helper code for the host for generating a LUT image in-memory to pipe through their image pipeline.

I disagree, this sort of implementation would be very host-architecture-specific. I would expect that any host wishing to use this functionality would already have a mechanism for constructing a LUT from the input and output of an entire render pipeline, not simply to replace one plug-in.

There was concern expressed at the TSC that this could be seen as circumventing OFX plug-in licensing if it were to be encouraged as a "buy only one licence, then automatically replace the plug-in with a LUT on all your other seats" feature. I think it needs to be framed as a way to use plug-ins in an existing "reduce this render pipeline to a LUT as best we can" workflow that [some] hosts already offer.

[gregcotten]

The criteria would seem to be: one input clip, at the current time (not multi-input, not temporal), where each output pixel's colour depends purely on [the parameters and] the colour of the input pixel at the same location (not an area operation like a blur, nor a position-dependent operation like a vignette).

I think it's important to note that plugins (like ours) often do have spatial characteristics applied, but still want to allow the user to generate a LUT by getting only the non-spatial characteristics of our effect. We currently have a built-in LUT generator in our plugin, but sometimes the user wants to be able to combine our effect with their own grading and bake that all into a LUT. That is still doable but requires many manual steps by the user to pull off correctly.

Passing a parameter (perhaps via the clip) stating that the plugin should disable spatial effects for the current render pass would be enough, I think.

I disagree, this sort of implementation would be very host-architecture-specific. I would expect that any host wishing to use this functionality would already have a mechanism for constructing a LUT from the input and output of an entire render pipeline, not simply to replace one plug-in.

Sure

There was concern expressed at the TSC that this could be seen as circumventing OFX plug-in licensing if it were to be encouraged as a "buy only one licence, then automatically replace the plug-in with a LUT on all your other seats" feature. I think it needs to be framed as a way to use plug-ins in an existing "reduce this render pipeline to a LUT as best we can" workflow that [some] hosts already offer.

I understand the concern, but my proposal is that this should be an opt-in feature for plugins.

In Resolve, for example, by default "no spatial awareness" is false and therefore the plugin will not participate in the LUT generation process. Also, even the laziest "bad faith" user could always generate their own LUT image and capture the color pipeline of an effect anyway.