Bring Your Own Component Registry Types

[kiersten-stokes]

Related issue: #2220

In order to BYO registry location type, the following 3 things must be implemented:

• Add a ComponentReader subclass to the component.py file
  • Subclass one of UrlComponentReader, FilenameComponentReader, or ComponentReader depending on where the component definition will ultimately be read from (remote url, local file system, proprietary catalog)
  • Implement the get_absolute_locations function
    • This function takes in a list of locations that each point to various component definitions, then loops through each of these locations to get a flat list of absolute locations to the component definitions themselves
    • e.g., for a Github-based registry, get_absolute_locations() could receive a list of links to Github repos and parse each of those links to get the direct links to all files in that repo (this assumes that all the files in that repo are component definitions)
      • (input) paths = ['repo1_link', 'repo2_link', ...]
      • (output) absolute_paths = ['repo1_component1_link', 'repo1_component2_link', 'repo2_component1_link', ...]
  • Define a location_type variable
  • Define the property resource_type that is defined as the location_type of it's superclass (you can use the example here exactly as-is)
• Add the reader to dictionary in get_reader (component_registry.py) like so:
  • [YourType]ComponentReader.location_type: [YourType]ComponentReader(file_types)
  • Include the relevant import statement in component_registry.py
• Add the value of the location_type variable for your new reader to the component-registry schema (elyra/metadata/schemas/component-registry.json) location_type metadata field enum:
  • "enum": ["URL", "Filename", "Directory", "YourType"]
  • The value entered into the above field must be equivalent to the reader class location_type variable value once converted to lowercase (in this example, location_type = "yourtype")

In order to process components from these types of registries, both KFP and Airflow rely on some aspect of the value of the location attribute on the Component object (the value of which is currently determined by the ComponentReader class's resource_type attribute). The following must be taken into consideration for the existing built-in processors:

• KFP
  • Custom components are loaded using the kfp.components.load_component() to construct the factory function that ultimately will be executed to create a KFP ContainerOp object
    • This function takes url, filename, or text values as parameters (see here)
    • This is the motivation behind setting and passing the ComponentReader class's resource_type (which then becomes the location_type of the Component object)
    • We don't currently use the text='component definition...' option, but it would be easy enough to implement with minor tweaks by either saving the component definition on the Component object or re-parsing a location and it's corresponding definition on an as-needed basis
• Airflow
  • In order to render custom components in the DAG, the operator module name (parsed from the location as the name of the file minus the extension) is needed
    • This is then rendered as such: from airflow.operators.{{operation.module_name}} import {{ operation.class_name }} (where operator classname is parsed from the Component object id)
    • Clearly, there's currently lots of room for error in this approach since it's a lot less strict than KFP component considerations

[kevin-bates]

Thanks for starting this @kiersten-stokes. I think I now understand what is meant by Component Registry Types.

One of the goals for making things extensible is that folks don't need to modify our framework and I feel like we should probably do something similar here. Most of the above is still applicable, it's just that we'd likely make the readers entry-point based and, just as the ComponentRegistry schema dynamically determines its enum of runtimes, it would do the same to determine its enum of location types.

I also suspect that some reader implementations will require metadata that is not currently reflected in the component registry schema and, if users were to require their own schemas, it seems like they could do that via a SchemasProvider in the ComponentRegistry schemaspace. This could get a bit hairy though since these schemas would need to derive from the existing schema or have a way to reference it in theirs. Hmm - we could instead build something where the ComponentRegistry Schemaspace instance "validates" its schemas (beyond what the SchemaManager does already - i.e., ensure they are applicable to the schemaspace). This could be the way we enforce that a BYO component registry schema has the requisite categories, runtime, location_type and paths properties defined w/o having to derive or reference the other.

We'd also need to make the registry instance available to the reader - but it looks like we have the registry instances in our hands when we go to get the reader, so this could be passed to the reader's constructor, for example.

Just some thoughts. Looking forward to seeing this all come about.

[ptitzler]

Based on observations I've made while trying to build a first prototype for the Machine Learning Exchange there are a few things that stood out. I essentially created a sub-class of ComponentReader and implemented the methods:

• method read_component_definition:

  • to retrieve the component specification from the source (MLX) I need to provide multiple pieces of information:
    • the MLX server's API URL (stored in the path)
    • the unique MLX catalog key of the specification (which is obtained in get_absolute_locations)

  def read_component_definition(self,
                                catalog_info: dict.    # the metadata information stored in `path` and other `path` related properties
                                component_info: dict,  # metadata information produced by `get_absolute_locations` that enables this method to fetch the specification
                                ...
  

• method get_absolute_locations:

  • The current signature flattens the output (to a list of strings), which should be okay for files and URLs but breaks other component resources, where the relationship between a path and the associated component ids in the returned list needs to be retained.
  • Example:
    • path1 = http://host1 # -> component_a, component_b
    • path2 = http://host2 # -> component_c
      For read_component_definition to work properly, it needs to know which path to use to find the component (e.g. path1 for component_b). Therefore the return data structure of get_absolute_locations cannot be list(string), because the association is lost. In theory the "absolute_paths" information for each spec might also comprise multiple values (key_info1, key_info2, ...) that read_component_definition knows how to use to retrieve the specification from the remote resource:
      e.g. path1[info] + key_info1 + key_info2 = component_b

[ptitzler]

Following up on Kiersten's update to the initial post, I realized that byocr probably requires changes to the .pipeline file. Excerpt from a KFP custom node definition (irrelevant properties removed) for a Filename registered component:

          "id": "b128c67f-a498-452c-ac2d-1e50dcd24397",
          "type": "execution_node",
          "op": "download-file_DownloadFile",
          "component_source": "/Users/patti/elyra_dev/workspace/examples/pipelines/run-pipelines-on-kubeflow-pipelines/components/download-file.yaml",

Let's say a pipeline file with such a node is opened in the VPE. Elyra needs to:

1. determine whether the node's component [specification] is defined in the palette (and thus accessible via the registry)
2. know how to retrieve the component specification.

For clarity I am using the term 'catalog' onwards to refer to an entity (local or remote) that stores component specifications to allow for the distinction with the term registry that Elyra uses internally. I view the currently supported location types Filename, Directory, and URL also as catalog types, which are not any different from other types the community wishes to contribute. As such each catalog type uses their own means to locate and retrieve an asset. This kind of view would also help us address this portability issue, because a new (conceptual) catalog type can be used to references relative filenames. (Unlike for example the Filename catalog type, a catalog entry of this type in the Elyra registry would require a base directory to be defined as a property, which is then pre-pended to the relative filename that's stored in the .pipeline file.)

There are several potential issues trying to do that if there are may types of remote catalogs.
(1) Lets say the op key is used to determine whether the component spec is present in the local palette/registry. This [generated] key must be unique across all component catalogs to assure that the locally registered component is indeed the same component that was used in the environment where the pipeline was created. How the key is generated would have to depend on the component's location_type because different location types might use different key information to "locate" the asset. Fictitious example keys (shown for illustrative purposes):

• localfs:<path-to-file>
• mlx:<mlx-api-url>:<mlx-component-asset-id>
• http-anonymous:<url>
• http-auth:<secured-url>
  Note that it's possible for the same asset (component specification) to be stored in multiple catalogs. I believe Elyra should not try to determine whether a specification that was retrieved from one catalog is identical to a specification that was retrieved from another catalog instance of the same type or a different catalog type.

(2a) Additional "component_source" metadata might need to be stored in the pipeline file. The kind of metadata would vary by catalog type. In the initial example in this comment, this metadata was the path/file of a local file. For other catalog types it might be some of the key information I mentioned in my previous comment to give the user a chance to determine what kind (and which) catalog needs to be added to the registry to gain access to the component specification that the pipeline file utilizes. This metadata

• would not include any confidential information (password, api keys etc)
• not necessarily include all information required to locate the asset in the catalog (e.g. the catalog specific asset id) for those types of catalogs where the Elyra registry creates an asset inventory on the fly. The latter is important because Elyra can generate the op key for each asset in the catalog and perform the matching (see (1)).

(2b) The node properties view in the VPE needs to expose some of this information to the user since the registry entry likely includes only core information that is not sufficient for use outside of Elyra.

[bourdakos1]

I got a little lost on today's call and have some clarifying questions around the terminology registry/catalog...

If I understand correctly, currently, a user has the option to create a Component Registry which can contain a list of components. The list of component must all have the same:

• runtime (kfp, airflow, etc...)
• location type (Filename, Directory, URL, etc...)

When a Component Registry is added, the components in that list are made available in the pipeline-editor's palette.

Question 1: What happens behind the scenes? Are these components downloaded and stored somewhere? Or do we just track the retrieval information?

The goal is to allow a user to create their own location type like MLX.

Question 2: What is the actually problem that custom location type is solving? Is it that MLX doesn't serve the necessary component format and we need to convert it to something Elyra understands? Or is it to support pointing to an endpoint and that endpoint returns a list of components (instead of having to manually enter a list of components, essentially the Component Registry points at another registry)

[ptitzler]

Question 1: What happens behind the scenes? Are these components downloaded and stored somewhere? Or do we just track the retrieval information?

A component registry entry (to be renamed to component catalog entry or similar for clarity in subsequent releases) encapsulates the information that Elyra needs to retrieve component specifications from a provider (which we'll refer to as catalog). Elyra itself does not permanently store the specification (e.g. YAML file for KFP). Note that, depending on the location_type (future catalog_type), an entry is resolved to zero or more specifications. ("Zero or more" comes into play for the Directory type and other (future) remote catalog types that can store many specifications.)

Question 2: What is the actually problem that custom location type is solving?

Both your answers + 1 additional:

• "doesn't serve the necessary component format " - Yes. For example, the catalog might send the specification file in an archive, which needs to be unpacked before Elyra can use it. Each location_type (catalog_type) is associated with a connector (currently called Reader) that performs the necessary tasks of (1) querying the catalog for a list of available specifications (2) retrieving and pre-processing one or more specifications (that Elyra wants to make available).
• "Or is it to support pointing to an endpoint and that endpoint returns a list of components " -> Yep.
• It enables developers to add support for catalogs that Elyra doesn't support out of the box. (MLX is one example.)

[bourdakos1]

Components feels like packages and package management. I know byo catalog_type isn't something that most package managers have to deal with, but it might be helpful to look at other package managers as a metaphor.

I think Carthage is a good example of a decentralized package manager that is popular in iOS development. They have the concept of origins

Origins include binary, github and git

binary "https://my.domain.com/release/MyFramework.json"   // Remote Hosted
binary "file:///some/Path/MyFramework.json"               // Locally hosted at file path
binary "relative/path/MyFramework.json"                   // Locally hosted at relative path to CWD
binary "/absolute/path/MyFramework.json"                  // Locally hosted at absolute path

github "ReactiveCocoa/ReactiveCocoa"                                 // GitHub.com
github "https://enterprise.local/ghe/desktop/git-error-translations" // GitHub Enterprise

git "https://enterprise.local/desktop/git-error-translations2.git"

This seems similar to what we are trying to do, but they delegate the resolving of the scheme (url, file) to the origin (binary, github, git)

This might be a separate discussion, but there's some other package managery things we might want to think about, like versioning or a distinction between global and project components.

I know versioning might not be feasible, but I think project components would be useful.

It would be nice to add a component to a pipeline and not have it added to the global palette. It would also be nice to share pipelines with people without them having to manually add all the components that I used in the pipeline.

[bourdakos1]

This is unrelated and definitely a separate discussion, but in my opinion, it's a bit overwhelming right now to add components to the palette:

• you first have to click on a new button that represents pipeline components, this button isn't connected to the palette in anyway. If didn't already know how to add components, I don't know if I would find it...
• after that, you need to click another button to actually open the creation form
• then you need to fill out a lot of things

I'm not saying we should do away with the pipeline components page, but it would be nice to have a quick add button on the palette. We could then infer things like runtime type. And this is where the single textbox feels nice, because the only thing you need to do to add a component is just paste the identifier

[ptitzler]

I agree that the current workflow is not intuitive (for first time users) and could be shortened. I'm totally in favor of adding "shortcuts" (of whatever type) to the VPE that make things easier or hide certain aspects. I believe at this point we are just working around current canvas limitations (e.g. a refresh button can't be added to the palette slide-out) to get an initial version out.

[ptitzler]

The relative path "base directories" is an interesting case... I could see how it would be necessary to have another form item to configure this. Can you elaborate on the use-cases of this a bit more?

• User A has component specifications stored in his/her local file system in /user/home/usera/mycomponents/
• User A creates a pipeline using these components.
• User A shares pipeline with User B who has the same components stored in the local file system in /user/home/userb/mycomponents/

Currently we only support absolute directory paths, which render the pipeline not portable because the referenced paths don't exist in both environments.

With relative path support both users could "clone" (in an abstract sense and done outside of Elyra) the content of a remote repository that contains the [project] specific component specifications. This way both have the same artifacts in their file system in any location that works for each one of them. One possible example implementation: a team stores the "project workspace" (includes all dependencies, aka pipeline files, notebooks, components) in a source control system. Every user can check out the "project workspace" into a local directory location of their choice and things would still work if the components are added to the registry via a relative directory path catalog entry.

With support for relative directory paths the pipeline file's node metadata would only include two pieces of information: the spec reference is of type relativeDir (making this name up) and has a property that holds the directory path mycomponents. The VPE/registry would assemble the absolute path from the base directory (which is configured for the catalog entry) and this offset.

Note that I am using the Linux path separator in my examples but IRL Elyra would have to treat path separator chars as OS agnostic tokens and replace them on the fly with the appropriate character for the OS where Elyra is running.

[ptitzler]

I'm not sure if you mean "one input text box" as in the identifier being a single string verses the potential of it having multiple form items. Or if you mean one generic box that allows you to type a url or a path without explicitly choosing which type.

One text box which would accept (for example) any of

"https://my.domain.com/release/MyFramework.json" "file:///some/Path/MyFramework.json" "relative/path/MyFramework.json" "/absolute/path/MyFramework.json"      

I would prefer not have to think as much about choosing things and opt for mindlessly pasting stuff haha. As in having a default config_type being similar to Carthage's binary, where I can just paste in anything (url, relative path, absolute path) and it would just work. Instead of having to explicitly choose URL every time I paste a URL.

Only time can tell whether my guess that Filename and URL catalog types won't be used much is correct. Bulk load (via a relative directory catalog or remote catalogs) should be a lot more efficient and a lot less labor intensive because users would add a lot fewer of those catalog instances to the registry. For example, by adding a single catalog to the registry, hundreds of components would be "imported" at once.

[bourdakos1]

Hmm this relative path stuff sounds very messy. I can definitely see the scenario of having a project with components and pipelines in one repo, but that use case seems more suited to what I was talking about with local components. It feels wrong to me that you would have a global set of components, but it is dependent and relative to this specific project. I'll open a separate discussion to talk more in depth about local components