Infrakit `template` command different from `utils init`

[wongwill86]

It seems that processing template actions via infrakit template is different from infrakit utils init w.r.t. passing in vars.

For example:

{{ include (cat (var `filename`) `.txt` | nospace ) | jsonDecode }}

processes differently when using the commands:

• infrakit template --var filename=... target.json
  • include action is inside target.json.
• infrakit util init --var filename=... target.json
  • include action is inside the script specified in the InitScriptTemplateURL parameter of target.json

Example repo here

I would expect that both commands does the var replacement from the specified --var in the command line.

However, I observe that only the template command correctly does the var replacement and the utils init command results in an error:

include: open /home/ubuntu/{{var`filename`}}.txt: no such file or directory

Thanks!

[chungers]

Thanks for testing this out. Yes this is a surprising result, but there's a reason for it. I am glad you are finding this out and my explanations consist of two parts:

Part 1 - infrakit template

First of all, there are a couple of changes I made to your groups.json and imported.txt. Let me explain. Here is the diff of my changes:

$ git diff
diff --git a/groups.json b/groups.json
index 0fdd94f..71caeb3 100644
--- a/groups.json
+++ b/groups.json
@@ -1,16 +1,16 @@
-{{ $initScript := `file:///home/ubuntu/init.sh` }}
+{{ $initScript := cat (var `config_root` ) `/init.sh` | nospace }}
 [
     {
         "Plugin": "group",
         "Properties": {
             "ID": "test",
-            "Description": "{{ include (cat (var `filename`) `.txt` | nospace ) | jsonDecode }}",
+            "Description": "{{ include (cat `./` (var `filename`) `.txt` | nospace ) | jsonDecode }}",
             "Properties": {
                 "Allocation": {
                   "Size": 1
                 },
                 "Instance": {
-                  "Plugin": "instance-vagrant",
+                  "Plugin": "vagrant",
                   "Properties": {
                     "Box": "bento/ubuntu-16.04"
                   }
diff --git a/imported.txt b/imported.txt
index a32a434..c9fe27e 100644
--- a/imported.txt
+++ b/imported.txt
@@ -1 +1 @@
-1234567890
+"1234567890"

My changes:

1. I used {{ $initScript := cat (var `config_root` ) `/init.sh` | nospace }} instead of {{ $initScript := `file:///home/ubuntu/init.sh` }} so that the init script is always relative to a root path that can be specified from the command line. This means I can have your repo checked out in any workspace directory and infrakit to calculate the relative path to get init.sh as long as I set the --var config_root flag in the command line. The hard coded path /home/ubuntu/ would break on a workspace on the Mac.
2. When infrakit encounters the include action, it will compute the path to fetch the file if there isn't a file:// or https:// protocol in the URL. So , I used include (cat `./` (var `filename`) `.txt instead of include (cat (var `filename`) `.txt to tell infrakit to fetch the file relative to the input groups.json.
3. I changed instance-vagrant to vagrant. The vagrant plugin has been compiled/ embedded into the infrakit binary, and you can start it up by

infrakit plugin start vagrant

This will start the vagrant plugin and infrakit communicates with it via a socket named vagrant. This is the default name. You can have a different name (say instance-vagrant), by doing this:

infrakit plugin start vagrant:instance-vagrant

The format is plugin_kind[:plugin_socket_name]. The second part is optional. vagrant is a special keyword / kind that's been reserved for the Vagrant plugin. See pkg/run/v0/ for a list of kinds or do infrakit plugin start to see what's been compiled into the binary.

4. The file content of imported.text has been modified to be a valid JSON by enclosing the string in quotes ". This is because you used | jsonDecode in your include pipeline:

{{ include (cat (var `filename`) `.txt` | nospace ) | jsonDecode }}

means include the file at the path (var `filename`).txt, pipe it to a function jsonDecode which takes a JSON text as input and returns as a parsed JSON in memory. This is the same as calling Go's json.Unmarshal on some text read from (var `filename`).txt. If you just wanted to include the entire json string as-is, you can just omit the jsonDecode altogether. Make sense?

So this is the groups.json:

{{ $initScript := cat (var `config_root` ) `/init.sh` | nospace }}
[
    {
        "Plugin": "group",
        "Properties": {
            "ID": "test",
            "Description": "{{ include (cat `./` (var `filename`) `.txt` | nospace ) | jsonDecode }}",
            "Properties": {
                "Allocation": {
                  "Size": 1
                },
                "Instance": {
                  "Plugin": "vagrant",
                  "Properties": {
                    "Box": "bento/ubuntu-16.04"
                  }
                },
                "Flavor": {
                    "Plugin": "vanilla",
                    "Properties": {
                        "InitScriptTemplateURL": "{{ $initScript }}"
                    }
                }
            }
        }
    }
]

And doing running from the repo's root directory, the command infrakit template yields:

$ infrakit template --var filename=imported --var config_root=file://$(pwd) groups.json

[
    {
        "Plugin": "group",
        "Properties": {
            "ID": "test",
            "Description": "1234567890",
            "Properties": {
                "Allocation": {
                  "Size": 1
                },
                "Instance": {
                  "Plugin": "vagrant",
                  "Properties": {
                    "Box": "bento/ubuntu-16.04"
                  }
                },
                "Flavor": {
                    "Plugin": "vanilla",
                    "Properties": {
                        "InitScriptTemplateURL": "file:///Users/davidchung/exp/issue-infra/init.sh"
                    }
                }
            }
        }
    }
]

So now let's run

$ infrakit util init --start vanilla --var config_root=file://$(pwd) --var filename=imported --group-id=test groups.json 

and we get

EROR[10-20|16:46:43] error preparing                          module=util/init err="template: file:///Users/davidchung/exp/issue-infra/init.sh:2:3: executing \"file:///Users/davidchung/exp/issue-infra/init.sh\" at <include (cat (var `f...>: error calling include: open /Users/davidchung/exp/issue-infra/{{var`filename`}}.txt: no such file or directory" spec="{Properties:<nil> Tags:map[] Init: LogicalID:<nil> Attachments:[]}" fn=github.com/docker/infrakit/cmd/infrakit/util/init.Command.func1
CRIT[10-20|16:46:48] error executing                          module=main cmd=infrakit err="template: file:///Users/davidchung/exp/issue-infra/init.sh:2:3: executing \"file:///Users/davidchung/exp/issue-infra/init.sh\" at <include (cat (var `f...>: error calling include: open /Users/davidchung/exp/issue-infra/{{var`filename`}}.txt: no such file or directory" fn=main.main
template: file:///Users/davidchung/exp/issue-infra/init.sh:2:3: executing "file:///Users/davidchung/exp/issue-infra/init.sh" at <include (cat (var `f...>: error calling include: open /Users/davidchung/exp/issue-infra/{{var`filename`}}.txt: no such file or directory

Specifically - there's this error: /Users/davidchung/exp/issue-infra/{{var`filename`}}.txt: no such file or directory. So you're right the --var filename=import somehow doesn't get to be passed to the template in the case of infrakit util init but it does in the case of infrakit template.

Part 2 infrakit util init

infrakit util init works differently from infrakit template. infrakit template applies a single iteration of the given template to render some body of text. infrakit util init however, is more complicated. It takes the groups.json definition and does the following:

1. It starts any plugins specified by the --start flag. In this case we have only the vanilla plugin because that's the only flavor plugin specified in groups.json.
2. It renders the input file as a template to do any interpolation necessary.
3. When the rendered JSON, it parses the JSON and based on the input --group-id it calls the vanilla plugin to Prepare the actual message that needs to be sent to the vagrant plugin for provisioning.
4. The flavor plugin is called to 'Prepare' the message to provision. The flavor plugin injects bootscripts based on what's specified in the InitScriptTemplateURL parameter. This value needs to be a URL that at this time can be resolved. This is a second rendering of a different template.
5. Once the flavor plugin successfully returns the config JSON, it extracts only the Init portion, which is the boot script. This makes it possible to do a infrakit util init groups.json | sh on a host that can become the genesis node that spawns a whole cluster -- because piping to sh means we are running the same script that infrakit would normally use to boot up the first node. This is the bootstrapping process.

You will notice that there are two iterations of rendering different templates in different places... the first time in your CLI process, the second time inside the flavor plugin. So the second time, it is run in a different context. This means the --var you passed in the command line wasn't passed along in this second evaluation. The scope of var is a single template evaluation, even if you include lots of templates. The second time the template is rendered by the flavor plugin it is a brand new context. This would explain why the filename variable isn't known by infrakit util init.

So this is a very long way to explain and to confirm that you've come across a limitation. I want to provide a detailed explanation in the comment so that there's documentation on this issue.

Solution

Because of this iterative evaluations of templates, that all use some common, user-specified global variable, we need a way to store variables like filename which can persist across processes and rendering of templates so that in this example filename is available inside the included script. This was raised by #646 and PR #716 was created for this reason.

So there is a work around, but it is not pretty... essentially this involves starting up the vars plugin, and commit the variable filename to it so that now filename is available globally for any plugin or any templates that call {{ metadata `filename` }}. If you also start up the manager, the variables will also be persisted in Swarm's raft quorum and be replicated to the other managers.

This workaround involves multiple steps of starting up plugins and adding values to it and then run infrakit util init, and it really isn't ideal. What we should be able to do is to be able to set the filename in the command line of infrakit util init and have that automatically added to the persistent metadata store so that you can just do everything in one line of code.

There are a couple of issues to work through but this is the basic idea for a future PR. I will submit that soon and thank you for finding this bug!