Criticisms of "dynamic plugins"

[jennydaman]

Introduction

So-called dynamic plugins (mentioned in some places such as 4-CUBE_env_to_remote.adoc) work like this: a plugin instance makes API calls to CUBE to wait on and schedule additional plugin instances.

Some applications of a dynamic plugin:

• wait on multiple existing plugin instances before creating a plugin instance
• run several instances of the same plugin, but with different parameters

For the most part so far, plugins strive to "work the same" when ran by CUBE or on their own. However, the separation of functionality which is CUBE v.s. a plugin's responsibility can be perceived as a blurry line. If "control logic/flow" (such as a feed join) were implemented in a plugin instead of in CUBE itself, it would demonstrate a way of extending the functionality and uses of CUBE without having to change CUBE.

Alternate Solution

An alternative approach would be to devise a specification or domain-specific language (DSL) for describing feed control logic which is interpreted by CUBE.

Criticisms

Access and Authentication with CUBE

The remote compute environment the dynamic plugin is running in may or may not have network access to CUBE. Hence, execution would work differently depending on which compute environment you choose. This is an issue to avoid because it would have reproducibility called into question.

Second, it is necessary for the dynamic plugin to authenticate with CUBE.

1. An inflexible solution is to hard-code the password.
2. An unsafe and inconvenient solution would be to require the username and password as an argument.
3. A good solution would be to implement a granular permissions model in CUBE where plugins may request permissions from CUBE. If we're going to be hacking CUBE anyways, why not implement dynamic plugins right then and there?

Inefficiencies

1. data egress
2. Wait for dynamic plugin to be scheduled by CUBE, then by the container orchestrator behind pman.
3. API is limited so the plugin would have to use busy waiting

Statefulness

The assumption that feeds are static can no longer be made. Static feeds are trivial to verify hence reproduce, whereas the graph of a feed containing a dynamic plugin is stateful.

For example, what if we were to try developing a tool which predicts the time needed for a feed to finish running. It is easy to imagine how to implement this tool under the assumption that the feed does not contain dynamic plugins. On the other hand, this tool would be wildly inaccurate for a feed with a dynamic plugin.

If there exists a specification for a schema or DSL for describing feed control flow which is to be interpreted by CUBE itself instead of a plugin, then we might be able to save this tool by having it consume the control description too (depending on what conditionals are supported by the spec).