POST and execute zapps directly (without a boot/system.map)

[larsbutler]

The documentation mentions that ZeroCloud is capable of executing "ZeroVM images" (tar archives containing a boot/system.map file) directly. Zapps, by comparison, are tar archives containing at least a zapp.yaml file, which contains a lot of the same information as system.map file.

It would be useful to be able to POST a zapp and execute it directly. To do this, there are a couple of problems that need to be solved:

• At the moment, we're tightly coupling application metadata with runtime information (such as file paths) in zapp.yaml. The solution I propose is to parameterize this and allow paths, etc., to be specified via query string parameters. For example: POST /version/account?input=swift://~/data/*.txt&output=swift://~/results/output.txt (with proper escaping/encoding).
• A zapp may have multiple possible "entry points". The POST URL should also a convention for specifying which group in the zapp.yaml should be executed. For example: POST /version/account?input=swift://~/data/*.txt&output=swift://~/results/output.txt&main=foo, where foo is the name of an execution group defined in the zapp.yaml.
• ZeroCloud would have to interpret these arguments and generate a system.map on the fly.
• Command line tools such as zpm would need a nice way of getting the arguments for these parameters.

[mgeisler]

Lars Butler [email protected] writes:

• A zapp may have multiple possible "entry points". The POST URL
  should also a convention for specifying which group in the
  zapp.yaml` should be executed.

I don't think this is consistent with the execution model in ZeroCloud.
The groups in a Zapp are all started at once and executed in parallel.

Or more precisely: what ZPM and ZeroCloud call groups are really
expanded into a list of zerovm instances that need to be executed. These
individual instances are executed in parallel. There is no well-defined
entry point in such a computation, and there are also not necessarily a
well-defined terminal node in the graph.

[larsbutler]

Actually, ZeroCloud has currently no awareness at all of the zapp.yaml. All it consumes is a system.map derived from the zapp.yaml. What I'm proposing is to stop putting so much "runtime" information (files, etc.) in the zapp.yaml, stop putting boot/system.map files into zapps, and let ZeroCloud generate it.

So, yes, it is not consistent with the current execution model. That's beside the point. What I'm suggesting is to change that model.

The groups in a zapp are only executed in parallel because the system.map defines it so. Technically, you can submit any system.map you want and execute anything inside a zapp. For example, you can execute any .py file in the zapp. What I'm suggesting is to turn the execution groups into well-known "entry points" (think of it like a public API) for the zapp, parameterize them, etc.

The problem I'm trying to solve with this is the tight coupling between the zapp.yaml and system.map, particularly the coupling between metadata (app-specific information) and runtime time information. This is preventing us from having a clean and consistent execution model.

[pkit]

We can make it work, but I would rather avoid it.
Placing more logic into proxyquery undermines the ability to run any executable code on the platform.
I.e. if you can already run any code and it can call any other code (since chaining was introduced) you can build arbitrary complex execution models.
I also think that the world is moving from templates into "everything is a code" paradigm, and I really like it. Mixing templates and code was never going to work. I'm considering removing the ability to template execution based on content_type that we have in /open/ endpoint, because it can be arranged by a proper chain.