Build from source in Vagrant

[diracdeltas]

Just a start; you probably want to define two Vagrant configs for install.sh vs install-dev.sh using config.vm.define so that non-developers don't have to build from source.

Warning: not extensively tested.

[garply]

Hey admins! Can one of you let me know if it's safe to build this pull request? See https://alpha.sandstorm.io/grain/kB7bofATTL2jGuKSFwmqdA/GithubPRCommands for a list of commands.

[paulproteus]

Garply, ok to test

[paulproteus]

Hi @diracdeltas ! (-:

I'm not sure what the default should be here! Or what the best practices are for the Vagrantfile handling both these cases.

One possibility is to use an environment variable to dispatch between these. Another is to create a Vagrantfile.dev and tell developers, "You can run:

vagrant -f Vagrantfile.dev

Then again, I don't think Vagrant actually supports that.

Given that, I think the thing I'd be happiest with is presuming that most people using the Vagrantfile are "just" developing apps, not developing Sandstorm itself, and telling Sandstorm core hackers to do:

HACK_ON_CORE=yes vagrant up

and let the Vagrantfile dispatch based on that environment variable.

Anyway, I'm curious what you think. And thanks for the patch!

[paulproteus]

I'm also surprised you don't need to apt-get install build-essential, but if you say this works, then that's great.

I'm also also curious what git is needed for. I see you added it in the README in this commit; I'd love to fully understand what makes it needed.

Also I asked @garply to test it but alas, Garply doesn't currently test the Vagrantfile, so it's sort of just symbolic in this case. (-:

[diracdeltas]

You can use config.vm.define in the vagrantfile to have two provisioning options I think: http://docs.vagrantup.com/v2/multi-machine/

I don't remember where git was needed; I think some part of the build involved git checkouts.

Sorry for brevity; replying on phone

[paulproteus]

​Now I get a different error. Paging @kentonv --

compile: sandstorm/spk.c++
/ekam-provider/canonical/sandstorm/spk.c++:33:10: fatal error:
'sandstorm/package.
capnp.h' file not found
#include <sandstorm/package.capnp.h>
^
1 error generated.
compile: sandstorm/sandstorm-http-bridge.c++
/ekam-provider/canonical/sandstorm/sandstorm-http-bridge.c++:45:10: fatal
error: '
sandstorm/grain.capnp.h' file not found
#include <sandstorm/grain.capnp.h>
^
1 error generated.
capnpc: sandstorm/ip.capnp
sandstorm/grain.capnp:23:38-48: error: 'GenericPersistent' has no member
named '
persistent'
sandstorm/grain.capnp:44:31-41: error: Not defined: persistent
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
capnpc: sandstorm/hack-session.capnp
sandstorm/grain.capnp:23:38-48: error: 'GenericPersistent' has no member
named '
persistent'
sandstorm/grain.capnp:44:31-41: error: Not defined: persistent
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
capnpc: sandstorm/supervisor.capnp
sandstorm/grain.capnp:23:38-48: error: 'GenericPersistent' has no member
named '
persistent'
sandstorm/grain.capnp:44:31-41: error: Not defined: persistent
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
sandstorm/supervisor.capnp:59:37-47: error: Not defined: Capability
install: node-capnp/node-capnp.ekam-manifest
node-capnp/capnp.node: not found
compile: node-capnp/capnp.cc
/ekam-provider/canonical/node-capnp/capnp.cc:29:10: fatal error: 'node.h'
file not
found
#include <node.h>
^
1 error generated.
compile: sandstorm/supervisor.c++
/ekam-provider/canonical/sandstorm/supervisor.c++:59:10: fatal error:
'sandstorm/
grain.capnp.h' file not found
#include <sandstorm/grain.capnp.h>
^
1 error generated.
capnpc: sandstorm/web-session.capnp
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
compile: sandstorm/run-bundle.c++
/ekam-provider/canonical/sandstorm/run-bundle.c++:25:10: fatal error:
'sandstorm/
package.capnp.h' file not found
#include <sandstorm/package.capnp.h>
^
1 error generated.
install: sandstorm/sandstorm.ekam-manifest
sandstorm/run-bundle: not found
capnpc: sandstorm/grain.capnp
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
sandstorm/grain.capnp:23:38-48: error: 'GenericPersistent' has no member
named '
persistent'
sandstorm/grain.capnp:44:31-41: error: Not defined: persistent
sandstorm/grain.capnp:83:27-37: error: Not defined: persistent
sandstorm/grain.capnp:161:54-64: error: Not defined: Capability
sandstorm/grain.capnp:543:63-73: error: Not defined: Capability
jstest: node-capnp/capnp-test.js
full log: tmp/node-capnp/capnp-test.js.log
capnpc: sandstorm/package.capnp
sandstorm/grain.capnp:23:7-17: error: Type names must begin with a
capital letter.
sandstorm/grain.capnp:23:38-48: error: 'GenericPersistent' has no member
named '
persistent'
sandstorm/grain.capnp:44:31-41: error: Not defined: persistent
compile: sandstorm/union-fs.c++
In file included from /ekam-provider/canonical/sandstorm/union-fs.c++:17:
/ekam-provider/canonical/sandstorm/union-fs.h:23:10: fatal error:
'sandstorm/package.
capnp.h' file not found
#include <sandstorm/package.capnp.h>
^
1 error generated.
make: *** [tmp/.ekam-run] Error 1
vagrant@localhost:/vagrant$

[paulproteus]

I can confirm git is used to download ekam and node-capnp etc. during the initial bootstrap phase.

As a note to myself, make update-deps (thanks to @kentonv ) fixes the issue I saw above. Also as a note to myself, git clean -f -d -x != git clean -f -f -d -x. Oh, well.

w/r/t http://docs.vagrantup.com/v2/multi-machine/ aka multi-machine, it looks if we are thoughtful, we can make the interface be:

vagrant up
vagrant ssh

etc., for a setup where we enable app development, and

vagrant up core-dev
vagrant ssh core-dev

for a setup where we enable core development.

It'll be a little clunky I fear, since the point of multi-machine is that Vagrant actively brings up all machines that are listed. For that reason, I vaguely prefer either:

• A flag file that acts a config option (like "touch .vagrant/SANDSTORM_COREDEV_MODE") that would cause "vagrant up" to, in effect, act in coredev mode, or
• An environment variable that causes Vagrant to bring the machine up in core dev mode, so you would do: CORE_DEV=yes vagrant up

It would only really matter for initial bring-up, I believe, anyway.

If we want the coredev-mode Vagrant to be on a different port -- which iirc we are already considering for the "regular" installer anyway -- then multi-machine mode seems best.

I'll sleep on it, but wanted to serialize my brain state here.

[diracdeltas]

@paulproteus You can tell vagrant not to bring up some of the listed machines; see the 'autostart' section of http://docs.vagrantup.com/v2/multi-machine/, assuming that's what you meant.

SecureDrop has an example of a multi-machine Vagrantfile that uses this feature: https://github.com/freedomofpress/securedrop/blob/develop/Vagrantfile