[Proposal]_Pinning_File_Suport

[Lypsolon]

Summary

• Implement a mechanism to specify an optional File that holds mapping pairs to overwrite resolution.
• This system should also implement an option to remove or not initialize the AyonCppApi so that communication with the server is impossible until you connect to a new Runtime cache that has an activated AyonCppApi

Glossary of Terms

Pinning

• a key-value pair from Ayon::Uri to Rootless Path
• the idea is to store the information that usually would be returned from Ayon-Server so that we can skip requesting the information

Ayon::Uri

• this is the Uri we can give to the Ayon-Server resolve endpoint to get back a single valid path. (it must be said that this usually also includes multiple returns like using versoin=*, but for this Proposal, we don't need this option, so we cut it from the definition)

Rootless Path

• this is the path Ayon-Server will return when resolveRoots is False

resolved

• this will be a fully resolved path so that USD can load it

cache Entity

• this will be the Struckt that lives in the runtime cache and holds all the information about the specific Ayon::Uri and resolved path that is needed to cache the resolution efficiently

path

• a system local path. (this is assumed to be correct on the system but does not necessarily need to exist; e.g., a shared drive might not be mounted, so the path would be correct but does not exist. But something like a Linux path on a Windows system would be false)

freezing resolution data in time

• The idea is that, e.g., latest or hero will not resolve to the newest version but to a specific version we store at a point in time.

Render farm/job

• in this proposal, I will use this context to describe all types of distributed jobs, not only Pixl Calculations. This includes but is not exclusive to (rendering, simulation, distributed model changes)
• the important part is that the work is distributed to a scalable system. It is not important if 1 or 100 nodes work on the task, but we need to assume that the system is scalable

update assets

• this will be the cover word for everything that might change the resolved path for an individual UsdAsset.

outside of the production network

• this will cover every system that does not have access to the Ayon-Server instance that handles the project the resolver is resolving for.
• this could be but is not limited to (render farm; has the files to render but might be on AWS and Ayon-Server instance is In local network, Local Render farm without access to Ayon-Server)

Conceptual Framework

Conceptually, the Proposal boils down to the following:

• a screen is ready to render
• we will write out the RuntimeCache to disk while we publish the scene. (currently, the assumption is that the runtimeCache will hold every Ayon::Uri as it is connected to stage Construction and not to something like the Usd::Hydra lazy interface, but this might need to be proven.)

Description

The abstract target for this feature is as follows:
We need to provide the Ayon_Usd_AssetResolver with data about what Ayon::Uri should be resolved into what path.
This is important for 2 Reasons

• freezing resolution data in time so that e.g a render farm/job does not update assets to eg, the latest while rendering
• making resolution possible without having access to the AYON server so that we can save bandwidth and allow rendering outside of the production network

Approval

Human:

• antirotor (needs to sign off as head of pipeline)
• dee-ynput
• m-u-r-p-h-y (it might be good to have input from Murphy to know if external pipeline teams will have an easy time working with this in case they want to implement their system around it)

External Implementations:

• Ayon_Usd sample case Implementation
  • full ayon project usage
  • USD should implement the following (Udim, Varients, Hearo and Latest Versions, Layers, References, Layers with stronger opinions about a reference to an UsdObject eg; LayerA objA=uriA LayerB objA=uriB, time sampling infos eg; animated variant changes(I'm not sure if this is supported by Usd needs testing), or Udim changes )

To Reproduce (if issue/bug/etc)

No response

User Stories

• As a Farm User, I want to be able to render a screen without changing the assets, re-render the scene, and get the same output.
• As a Render Wrangler, I want to distribute a render job to 1 or 1 million machines without slowing down the Ayon-Server at all; I also want to be able to render with a system that does not have access to my Ayon-Server instance

Impact

High

Assumed Complexity

Days, Months

Impact Scale

an entire tool group

Other tools that get touched

All Dccs that have Ayon_Usd suport will be affected by this.

How it Solves the Problem

Allowing to load a file at startup will allow us to use the resolver without having to access the server;

• This will fix Many resolvers trying to resolve simultaneously and sending duplicate requests to the Ayon server.
• Not accessing the server and storing the data in a file also means that no changes will be automatically made after writing the pinning file.

Implementation Idea/Details

Resolver Caching and CppApi

• Every Resolver will automatically connect to the Global runtime cache instance.

• We will check at startup of the Global Runtime cache to see if a pinning file is present via an environment variable, and if this is the case, we will not initialize the cpp API. (we should have an extra env variable to disable this behaviour without having to delete the pinning file)

• Then, we will load the file. And generate the local path

  • [issue] Somehow, we need to find out what the local root_replace paths are because they could be different, and the pinning_file should only save {root_lees} as we can't know what the root path will be on this system
    • we could expose them as env variables when we start the task. (this could limit the ease of use a good bit tho)
    • We could store the name/identifier for the system with a list of root replacement variables in the file. (this might be hard to know beforehand)
    • We could enforce that if you want to use this feature, you need to have a file defining those root replacement paths (this is inferior to env variables, I believe because it adds more complexity)

• And write it into the cache

  • [optional] For this, we will reduce the multiple vectors and create one big vector that holds all the Cache_Entities
  • we will take every object in the cache and replace the root (this will need a new function because usually the AyonCppApi handled this rootReplace) so that we get the full path and generate cache_entetys from that
  • The cache entity struct will have a function for re_eveluate, and we will add a key to the Cache_Entety Statick, which will then always block re_evaluation.

This approach has the disadvantage that we assume that we will have all the needed paths and don't want to talk to the server, and while this is the point of this proposal, we should consider if there are edge cases that break this assumption

Technical Approach

Storing the cache in a file

writing out the file will require the following steps.

• take all the cache_entetys and the root paths and "un_resolve" them so that a full os_Path will end up being a rootles_Path again (for this, we will need the specific root replace option used to generate the cache_entety add_hanlde).
• then we can take the Ayon::Uri and the rootles_path and simply store them like a pair()

as a USD

• in USD, we can add metadata to a .USD file via the GetCustomLayerData (the data type could be VtDictionary

as JSON

• In the case of JSON, we could use a JSON dictionary and Ayon::Uri as the key.

Loading the File

as Usd

• in USD, we use the Sdf::Layer FindOrOpen system to load the single file without constructing a stage GetCustomLayerData will allow us to read in the data directly from the layer.

as JSON

• In JSON we can use the nlohmann::json library to open and read the JSON dict.

We need to specify a standard in both cases, probably in a class. (we could write a class that takes all the info and has the write read operations as member functions so it can enforce the data layout on disk)

Architecture

No response

Unresolved Questions

• Does writing out the Runtime Cache include all Ayon::Uris needed to render the scene?

  • are things like Udims, Varients, Out of Thrustom objects resolved in all cases
    • UDIMs are not resolved at stage open _ResolveUdimPaths is handling this on request.
  • can we force the whole scene to be resolved, and if so, will this cause unnecessary resolutions

• if we decide to remove unused cache entries after a while. How does this affect Writing out?

  • it is currently unclear if we will remove Cache Entities after a TTL or mark them as "Re-Resolve-Please."

• Should the Resolver write out all the Ayon::Uris, or could we have a separate system for this?

  • taking the ayon::Uri from the Usd file is entirely pointless because we would need to re-resolve it, and that would potentially update an asset if, e.g. the artist published to rendering and, during the scene gets prepared, another artist pushed an update

milestones

• test if writing out the resolver cache will be enough (regarding Udims, time samples etc.) (if this is not the case, create a request for it, as the creation of the pinning file is not blocking this issue and the file structure will be defined by loading not by writing)
• [Feature Request]_Rework_assetIdent #18 <- Implement this issue so that we can block re_resolve for an cache_entry
• implement a class for writing + reading data from files (needs strict structure enforcement)
• decide on a solution for obtaining the site Root overwrites (@antirotor and or @dee-ynput would need to green-light the decision)
• implement env variables for pinning file and enable pinning
• abstract the rootReplace function into a generalised function and move it to cpp-dev-tools /// or create a function under the resolver and create a task to move it into cpp-dev-tools
• create tests for loading just from pinning_file. (needs: refs, sublayers, Udims, variants, animated assets, (maybe) animated textures or other references)
• build a test workflow to test with ayon_distributed systems (eg: aws_deatline)

Potential Problems

• this change might cause quite a bit of refactoring as the Ayon_Cpp_Api attachment to the resolver cache was not built with this in mind

Long-term Issues

No response

Transient Problems

• depending on the chosen implementation, it might be needed to fully utilise ayon-cpp-dev-tools and move forward with the construction and implementation of the toolset into the dependent tools (cpp_api)

impacting Branches

• ayon-cpp-dev-tools (it might be a good idea or needed to finish up the baseline implementation of this toolset and implement in the needed spaces as this would be the cleanest option)
• ayon-cpp-api (might need to implement dev tools as well and could move some functions into dev_tools)
• ayon-usd should implement functions to utilise the feature.

Dependencies

No response

outgoing Dependencies

No response

dependency Issues

#18
ynput/ayon-usd#17

Dependant Branches

No response

Common Dependencies

No response

non-persistent test (e.g. development time tests)

• The tests to run with AWS think box will probably need to be setup by hand for now as we have no way to auto-up that system with Ayon etc

Testing Plan (president tests)

• build a usd file that has all the things we want to test in it, and then load it, write out the pinning file, create a new Ayon_Usd_Standalone instance and load the USD file via the pinning support system, then compare the stages

Documentation

• add documentation about how to use the env variables
• add documentation on when and where to use the feature

Maintenance

I assume the feature will require little maintenance after implementation, but new USD features or requirements, changes, and requests for different usage methods could lead to manual rework.

Risk Management

the usage of the tool should only be advised after extensive testing, and it should stay out of production until extensive testing.

Furthermore, disabling the feature would still allow using the resolver, but if we want to propose it for a render farm, this feature rollback would not be an option.

Appendices

No response

Is there an existing issue with this?

• I have searched the existing issues

Are there any labels you wish to add?

• I have added the relevant labels to the bug report.

[Lypsolon]

im gonna throw this this link here:
its a talk between BigRoy and some other Usd users about stage traversal and getting paths. (might be useful for generating the pinning file in the end)
https://academysoftwarefdn.slack.com/archives/C013Z5AMT7T/p1707348121426559

[Lypsolon]

We do it in JSON

[Lypsolon]

Usd utils have a function to get all files across the stage

[BigRoy]

Usd utils have a function to get all files across the stage

This topic should provide you some ways to do it:

• https://forum.aousd.org/t/get-all-references-sublayers-recursively-of-stage/1322/3?u=bigroynl

[Lypsolon]

$Site_Root_Overwrite can be an environment variable and Root replace can use it

[Lypsolon]

Linking this GH issue for further reference.

This issue covers the changes in behaviour for the UsdUtils.ExtractExternalReferences function between USD versions.

We need to decide if it is safe to use the function or if we should build our own setup to extract the paths for the PinningFile

PixarAnimationStudios/OpenUSD#3173

[Lypsolon]

this has been finished with the Feature/23 feature request pin file loading merge