Proposed Basic Color Management

[reinecke]

Background

OpenTimelineIO is used to communicate compositional information starting from "dailies" authoring, all the way through conform. OTIO has the opportunity to preserve color decisions starting with those made on set (e.g. LUT application, CDL values, etc.) all the way through to finishing.

The objective of color management in OTIO should not be to advocate any particular color workflow, pipeline, or mathematics - it should be to convey the color pipeline and decisions used when authoring a composition.

Explicit vs. Implicit Transformation

Color pipelines can be expressed in two different ways:

• Declarative transforms - Color spaces at various target scopes (source media, timeline, output space) are specified and it's left to the implementation to determine how to transform between them
• Imperative transforms - Specific transformations (LUTs, CDL values, etc.) describe how content's color transforms through the pipeline

Color In OpenTImelineIO

In OpenTimelineIO, I propose we use a combination of declarative and imperative models so that the focus is on what color transformation should happen in at which points and only get into how to perform those color transformations when necessary. This delegates the decisions about "how" to the discretion of implementers - OpenColorIO can be suggested as an implementation for resolving from declarative to imperative color pipelines.

Declarative color pipeline can be achieved through adding a color_space string attribute (values in this attribute to be discussed later) to various objects in the timeline hierarchy which identifies colorimetry for specific checkpoints in the hierarchy.

The proposed attributes would be:

• output_color_space on Timeline to describe what color space the rendered output of a timeline should be (i.e. what colorspace a file on disk would be written in)
• Add a working_color_space attribute to Timeline to specify the colorspace timeline transform operations should be done in
• Add a nullable color_space attribute to Composable to specify the color space transform operations should happen in at that point in the timeline heirarchy
• Add a color_space attribute to MediaReference (that can be null) to state what the colorspace of the input asset is - this should ideally match metadata in the source file, but could be used to override when sources don't provide it or have the wrong metadata
• Add a base ColorEffect with a working_space attribute. If null, the working space is presumed to be same as the Timeline working_space, otherwise, this is the color space the transform should be applied in
• Add a CDLColorTransformEffect subclass schema for encoding ASC CDL values (see Foundry CDL Effect Proposal #1793 for more info)
• Add a LUTColorTransformEffect subclass schema with a lut_reference attribute of MediaReference type for identifying an external LUT file to use

To determine the color space of any element in a composition, check the element itself for color space metadata. If it's not there, check it's parent, and it's parent, and so on, all the way up the hierarchy until a color space metadatum is found. If none is found in this manner, the default color space should be Rec. 709, indicating the historic srgb transfer function, the most common default rec709 primaries and whitepoint, and scene referred, indicating that there is no other color management role implied.

This allows the Timeline to specify the output color space and the , individual media to be associated with the proper input color space, and clips to have specific transformations and the colorspace those transformations should be performed in.

For example, a typical ACES pipeline (pulled from AMF example3) might be setup as follows:

# Create the timeline and set our output and working spaces
my_timeline = otio.schema.Timeline(tracks=[otio.schema.Track(name="V1", kind=otio.schema.TrackKind.Video)])
my_timeline.output_color_space = "P3-D60 - Display"
my_timeline.tracks.color_space = "ACES2065-1"

# Create a reference to the source media and specify the input color space
source_media_reference = otio.schema.MissingReference(name="A001C030")
source_media_reference.color_space = "Acme Camera"

# Create the clip and add the imperative color transforms
clip = otio.schema.Clip(name="A001C030", media_reference=source_media_reference)
clip.effects.add(
    otio.schema.CDLColorTransformEffect(
        name="Technical Grade",
        working_space="ACEScct",
        slope=[2.0, 2.0, 2.0],
        offset=[0.1, 0.1, 0.1],
        power=[1.0, 1.0, 1.0],
        saturation=1.0,
    )
)
# On this transform we leave working_space=None because it matches the timeline working space
clip.effects.add(
    otio.schema.LUTColorTransformEffect(
        name="Acme DI Show Look",
        lut_reference=otio.schema.ExternalReference(target_url="urn:ampas:aces:transformId:v1.5:LMT.ACME.AcmeDILook.a1.v5"),
    )
)

Color Space Names

To ensure consistent interoperation across applications, there should be consistency about how string values map to the specific colorimetry definitions for a color space. From my research, there seems to be no such canonical table. For the domains OpenTimelineIO typically serves, the OpenColorIO Configuration for ACES seems to be furthest along this path.
This draws from both ACES library and OpenColorIO imperative transforms, maps some attributes/naming through a table, and creates an OpenColorIO config that can be used to find an imperative transformation between declarative color spaces.

While the ACES library defines keys (ACESTransformID URNs) to identify imperative transforms, the ACES OpenColorIO configs define a list of declarative color space names.

Open Questions

• Should the various color space attributes be nested inside of something like a "visual attributes" structure to avoid having audio, video, and text attributes clouding the same namespace in the objects?
• Are CDLColorTransformEffect and LUTColorTransformEffect too verbose?
• Which fields from the OCIO config should be used? The name field is designed to be human friendly, one of the options from aliases tend to look a bit more like a good key to encode in a serialized file.

Related Issues/Discussions

• CDL color effect proposal: Foundry CDL Effect Proposal #1793
• Project settings issue: Add Sequence/Project Settings to Timeline #1683

[meshula]

This looks like a great start; I have a question :)

Does the lut indicated in LUTColorTransformEffect explicitly transform to the working color space indicated on the BaseEffect? I believe the answer is yes, but just want to confirm. I assume the same is true for the CDL effect.

[reinecke]

I think this aligns with what you said, but this is my idea what the resolved imperative color pipeline steps in the above example would be to process the clip for display:

1. Clip pixels are loaded from file
2. Pixels are transformed using an implementation resolved Acme camera (Media Ref Colorspace) -> ACES2065-1 (Timeline working space)
3. CDL Transform Effect is applied in the following steps
   1. Pixels are transformed using an implementation resolved ACES2065-1 (Timeline working space) -> ACEScct (effect working space)
   2. CDL transformation is applied with effect provided values
   3. Pixels are transformed using an implementation resolved ACEScct (effect working space) -> ACES2065-1 (Timeline working space)
4. Apply LUT effect
   1. No effect working space is provided, effect working space assumed to match timeline, no-op
   2. Apply transform described by reference LUT
   3. No effect working space is provided, effect working space assumed to match timeline, no-op
5. Pixels transformed using implementation resolved ACES2065-1 (Timeline working space) -> P3-D60 - Display (Timeline output color space)

This is an example of how I imagine this would be resolved. I'd imagine that some implementations might do some "under the hood" optimizations - for example if there are two consecutive effects in the same working space they could be applied in succession without going in and out of timeline working space in between. However these kinds of things are up to the implementer based on their domain and knowledge of what implications that may have on the quality of the output and other factors - technically the OTIO would be describing a color pipeline that does these potentially lossy steps between each effect.

EDIT: Given the proposed removal of timeline working space in the subsequent comment, the lookup on what to use as timeline working space would be slightly different (using the color_space resolved up the hierarchy for the Track) but the result is the same.

[doug-walker]

A CDL is typically intended to leave the pixels in the original color space. However, that is typically not true for a LUT. A LUT could be just a look but it could also be designed to convert the color space. You might want to consider these OCIO transforms as a taxonomy of the sort of things that might be useful:

FileTransform -- Apply a variety of color transforms (ASC CDL, LUT, etc.), without converting to a process_space. Given that a LUT may change the color space, it would be helpful to include a color_space attribute to indicate what it converts to. (OCIO has a related transform called NamedTransform which essentially is a FileTransform included, by name, in a config.)

LookTransform -- Convert to a process_space, apply a FileTransform (CDL, LUT, etc.), and convert back to the original color space.

ColorSpaceTransform -- A conversion between a pair of named color spaces that have the same image state (i.e., either scene-referred or display-referred).

DisplayViewTransform -- A conversion between a pair of named color spaces that have differing image states. Crucially, this must take a third argument (in addition to the input and output color spaces) which is the specific tone-mapping (or view transform, in OCIO parlance) to apply.

The ColorSpace and DisplayView Transforms are essentially the way to convert the declarative references to imperative transforms using OCIO.

[meshula]

Another question, should tracks themselves have a color space? I'm imagining a composition that uses one group of tracks to target say a projector, and another group of tracks to target a DVD. The timeline color space would then represent the color space to use for any elements in a composition that do not have a more local color space in their hierarchy.

Probably should say something about color space resolution, like,

To determine the color space of any element in a composition, check the element itself for color space metadata. If it's not there, check it's parent, and it's parent, and so on, all the way up the hierarchy until a color space metadatum is found. If none is found in this manner, the default color space is srgb_rec709_scene, indicating the historic srgb transfer function, the most common default rec709 primaries and whitepoint, and scene referred, indicating that there is no other color management role implied."

[reinecke]

This makes quite a bit of sense, I think putting an optional color_space attribute on the Composition schema object would certainly make the the color space resolution rules really clear like you describe, they would match the kinds of hierarchy traversal that are common in OTIO. I would advocate we include API to resolve this for implementers (e.g. something like track.color_spaces_to(child) that would give a list of the colorspaces to transform through in order when going from the child up through the timeline parent hierarchy).

If we did put color_space on the composition level, that would imply we can drop working_space from the Timeline level - the working space would be the color_space of either the timeline.tracks Stack object or resolved up the hierarchy to the timeline and finally defaulted as you described. I think this makes the mental model clearer because it says "all color operations at this level of the timeline hierarchy should be done in color_space" - and it is clear that anything down the hierarchy used at that level should first be brought into that color space.

The only concern I have is that the stated example where individual tracks might have separate visual outputs - I'm not sure this is conventionally supported in most NLE software. That said, the Timeline. output_color_space attribute would then tell that software a final colorspace to conform all those tracks to for its purposes. So that use case might fall in the "OTIO supports it, but NLEs may not support/preserve it".

[jminor]

I have doubts/concerns about the hypothetical use case "one group of tracks to target say a projector, and another group of tracks to target a DVD".

In a scenario like this a wide range of output-specific properties like resolution, frame rate, color space, etc. would be different between those two outputs. To support a use case like that we should either model an OutputSettings schema that bundles those, and/or state that separate Timelines should be used to represent the two uses of the same composition. Some verbal discussion in past OTIO TSC meetings revolved around the idea of including a list of named OutputSettings objects at the top level Timeline to support multiple rendered outputs for one composition.

The other extreme, in which every Composition (or every Composable?) specifies its color space might actually align more with the way OTIO models time ranges (available_range, trimmed_range, etc.) and spatial bounds. Adding color information at this granular level would allow varying color spaces at a clip-by-clip level as well as track-to-track. In the case where an OTIO models a simple playlist of multiple clips played back-to-back, having clip-by-clip variation could be really handy. For example, maybe I am playing back clips to specifically A/B compare two different color spaces. Is that a use case we aim for OTIO to support?

[reinecke]

In the current proposal, a Clip assumes the color_space of its parent. color_space could become an attribute at the Composable level. In general, I don't expect people would use this though it would provide a way to effectively set a shared default working_space for all the color effects on the clip (though color effects would also need to maintain their working_space overrides).

One interesting side effect of this would be that Gap would gain a meaningless color_space attribute.

My instinct says that setting color_space at the clip level is excessive, though I don't have any strong arguments against allowing it either.

[meshula]

So, I think Josh answered my question about whether tracks plus color space can model multiple outputs, the answer is no.

The idea that he suggests possibly falls from it, which is a color space can be settable on a Composable holds water for me. To Eric's point that assigning color space to some composables like a gap is meaningless, is well taken. From a data modeling point of view though, making it available on composable would certainly make traversal simpler than if it's applied to some composables and not others. Perhaps it could be general in this manner, but there could be explanatory notes in documentation along the lines of "setting the color space on a [gap, and what other things?] has no meaning".

[darbyjohnston]

Should the various color space attributes be nested inside of something like a "visual attributes" structure to avoid having audio, video, and text attributes clouding the same namespace in the objects?

Maybe a "color" attribute? Like:

my_timeline.color.output_space = "P3-D60 - Display"
my_timeline.color.working_space = "ACES2065-1"

Are CDLColorTransformEffect and LUTColorTransformEffect too verbose?

I think having descriptive names is good, especially if there are going to be more effects. Is "LUT" meant to refer specifically to a 1D LUT, or is it more general like the OCIO::FileTransform which can be any file based color transform?

Which fields from the OCIO config should be used? The name field is designed to be human friendly, one of the options from aliases tend to look a bit more like a good key to encode in a serialized file.

Which aliases do you mean? I thought aliases in OCIO configs were just alternate names for other color spaces.

[reinecke]

Is "LUT" meant to refer specifically to a 1D LUT, or is it more general like the OCIO::FileTransform which can be any file based color transform?

This is a good callout. It was something that I felt a twinge about when I included "LUT" in the name for exactly what you bring up - it's meant to communicate there is some black box description of the transform to apply that is external to OTIO.

I used "LUT" in the name because:

• I was following the Common LUT Format example - though you'll note that the majority of ProcessNode subtypes are in fact not LUT-based at all
• LUT is a more artist-friendly term

All that said, I like the idea of leaning toward something a bit more like OCIO's FileTransform. I shy away from "file" because the transform could be described as a something like a .clf, .ctl, or .cube - but it also could use something like an ACESTransformID URN that systems like Baselight can use to resolve a built-in transform standardized as a part of ACES.

For example, ACES specifies a reference gamut compression identified by a URN like urn:ampas:aces:transformId:v2.0:Look.Academy.ReferenceGamutCompress.a2.v1 and they provide a reference implementation as CTL, however vendors may opt to recreate a mathematically equivalent transform in a way more technically aligned with their implementations (e.g. as a shader program or similar).

Maybe something like ExternalColorTransformEffect properly captures the idea?

[darbyjohnston]

The Color Interop Forum has defined a first round of canonical snake_case aliases

Thanks, I didn't know the aliases were being standardized. Is there an associated .ocio file or do they map to the built-in OCIO color configs?

Maybe something like ExternalColorTransformEffect properly captures the idea?

That seems reasonable, though maybe LUT is fine since it is kind of a generic term now. I was thinking of it from a developers point of view in terms of what formats I would need to support. Interestingly the comments for OCIO::FileTransform also refer to the files as LUTs.

[reinecke]

The OCIO studio config releases include .ocio files with the named color spaces - it's mostly about aligning on the set of aliases that make the "basic canonical set".

[meshula]

As far as the CIF color spaces go, they will ship with OCIO itself, and always be available. I believe the current state of affairs is that those confs are in the nanocolor branch of OCIO, but they're intended to make their way to the main branch after the nanocolor branch matures a little.

[doug-walker]

Yes, the CIF color spaces (both the user-facing and compact names) are already in the OCIO CG and Studio configs for ACES that are included in the OCIO 2.4.0 library. There is also a smaller config that only implements the CIF spaces that is currently only available in the OCIO nanocolor branch, as Nick wrote.

[doug-walker]

The example above includes these lines:

my_timeline.output_color_space = "P3-D60 - Display"
my_timeline.tracks.color_space = "ACES2065-1"

So the track color space is scene-referred (ACES2065-1) and the output color space is display-referred (P3-D60 - Display). It's important to recognize that there are many of different transforms to convert between scene-referred and display-referred color spaces. For example, ACES alone has dozens of ways, digital cinema vendors each have their own renderings, as do post-production houses and cinematographers. It is essential to include this information when converting between image states.

This conversion goes by a lot of different names (tone-map, output transform, display-rendering transform, etc.) but in OCIO it's called a view transform and is applied via an OCIO DisplayViewTransform which takes three arguments: the scene-referred color space, the display-referred color space, and the view transform.

So if you're going to have an "output_color_space" attribute, I would strongly recommend a view transform attribute as well to encourage consistency among implementations. Perhaps better, you could avoid having an output_color_space attribute and instead define a DisplayViewTransformEffect that could be used to convert tracks to the desired output color space.

I don't know if Gap tracks are allowed to have effects on them, but that could be a useful way to harmonize a set of clips in different color spaces into a common color space if the Gap could contain either a ColorSpaceTransform or DisplayViewTransform.

I realize this is steering a bit towards allowing more imperative operators, but I feel that timeline authors will want that control. For example, transitions such as dissolves require adjacent clips to be in a common color space. So it may become important to have a way to harmonize clips that originate from different sources or processes.

[john-paulsmith]

OpenFX recently completed a project to add colour management for plug-ins, which used similar concepts to this proposal. We used the OCIO ACES Studio config as the source for most strings in the same way that you are thinking, which allows developers to implement the spec using any colour management system that supports ACES, and makes it trivial to implement using OCIO. Here are some of the key decisions we made:

• The OCIO ACES configs change over time, so it's necessary to include a version string to identify which schema the colourspace identifiers belong to. In OpenFX, we have a string similar to the OCIO config name: ofx-native-v1.5_aces-v1.3_ocio-v2.3, which includes the OpenFX version, ACES version and OCIO version. This is also the name of a procedurally-generated C header that includes all the string constants for that version of the spec.
• We defined some simple logic to select the internal string for each colourspace:

def colourspace_short_name(spc):
   name = spc.getName()
   aliases = spc.getAliases()
   if ' ' in name and len(aliases) > 0:
      return aliases[0]
   else:
      return name

• There are several different levels of colour management, to cater for the wide range of plug-ins and hosts implementing OpenFX:
  • Basic: a set of six generic colourspaces which are incompletely specified but are enough for a plug-in to approximately know how to deal with the clip. Our set is:

  'ofx_display_sdr': 'Any display-referred SDR video such as Rec. 709.',
  'ofx_display_hdr': 'Any display-referred HDR video such as Rec. 2100 HLG or PQ.',
  'ofx_scene_log': 'Any scene-referred colourspace with a log transfer function.',
  'ofx_scene_linear': 'Any scene-referred linear colourspace.',
  'ofx_raw': 'Image values should not be treated as colour, e.g. motion vectors or masks.',
  

  • Core: a set of colourspaces which corresponds to the OCIO ACES CG config, with the addition of PQ, HLG and P3 display spaces:

  ['srgb_display', 'displayp3_display', 'rec1886_rec709_display',
   'ACES2065-1', 'ACEScc', 'ACEScct',
   'ACEScg', 'lin_p3d65', 'lin_rec2020',
   'lin_rec709_srgb', 'g18_rec709_tx', 'g22_ap1_tx',
   'g22_rec709_tx', 'g24_rec709_tx', 'srgb_encoded_ap1_tx',
   'srgb_encoded_p3d65_tx', 'srgb_tx', 'Raw',
   'rec1886_rec2020_display', 'rec2100_hlg_display', 'rec2100_pq_display',
   'st2084_p3d65_display', 'p3d65_display']
  

  • Full: everything from the OCIO ACES Studio config except roles.
  • OCIO: in this mode, the host is expected to provide the path to an OCIO config, at which point any valid string from the config can be used. If OTIO is being used in a scenario where everyone has access to the same config, this could be a useful option.
• The standard roles from the OCIO config are not used as part of the spec.

References:

• Python script to generate a C header with all the strings used in the spec
• Latest generated header
• API with comments