Technology Stack

[knownasilya]

HTTP with streams?
HTTP long-poll?

RabbitMQ?
etc..

This is regarding core<->drone

[knownasilya]

@microadam primus looks very nice.

[phiros]

I'd also go for primus. Would allow us to switch the underlying real time framework later if we
need to.

BTW for reference: Travis uses a combination of pusher and amqp for this.
AMQP is for the communication between the worker (drone in your terminology) and
travis-hub (core in your terminology). The logs are streamed with pusher from the
worker to a logging component and to browsers.

[knownasilya]

Do you see using a REST and a socket API together, since not everything needs to be realtime?

[phiros]

Yes I think it makes sense to use the realtime communication channel (= the socket API) only when you really need it. In my opinion this is only needed for streaming drone output to strider and from strider to the browser. Everything else can and should be done via REST (job distribution / job cancellation etc.).

[knownasilya]

Agreed. And with Hapi's injection API we can make it all REST, and then reuse the REST endpoints with the sockets. It's a win-win.

[davemackintosh]

@knownasilya Hapi's injection API? You mean Shot?

[knownasilya]

Yeah (still figuring out how Hapi is architected). I'm not sure about this yet. Would also love for examples on how to structure a Hapi app.

[davemackintosh]

I might have something of interest here in the next couple of days. What are your thoughts on API authentication and auth in general for Strider?

[knownasilya]

Token auth with API tokens for external resources like drones.

[davemackintosh]

JWT? Or something home-cooked?

[knownasilya]

Yes, JWT.

[davemackintosh]

Ahh cool, let me think. I might have something that will make this 1000X easier and faster based on Hapi

[zakjan]

Is there any reason for JWT instead of plain random tokens?

[knownasilya]

JWT can carry data along, which is super useful for things like drones and plugins.
Plus there is no reason not to use them really, since both are tokens and have the same cons.

Plus you can limit them by recipient and have an expiration time.

[knownasilya]

I'm not so sure about seneca microservices. They seem to complicate everything quite a deal for almost no benefit. Core will run on one machine and so far there are no plans to split it out, nor would there be any reason to. It seems like a pre-mature optimization that hinders readability, understandability, and limit the number of contributors that could come in and start contributing.

[davemackintosh]

@knownasilya RE Seneca. It's awfully convoluted for something as small as this, until core is split across multiple systems and does many different things simultaneously it would over complicate and dissolution core contributors and open source contributors.

However much I love NearForm this project isn't mature enough to expand with micro services, that's pretty much what the drones are for IIUC

[phiros]

We don't necessarily need microservices to structure core itself.
However, I still find it a worthwhile endeavor to use microservice for three reasons:

1. It allows us to prototype functionality without the need to think about whether this
   functionality should be a plugin or not. Microservices can be very easily moved into
   a plugin if need be (my simplistic github provide would be a candidate for this).
2. Seneca is essentially a (backend) plugin API which we don't have to write.
   Registering a plugin is as easy as starting the corresponding microservice.
3. It allows us to isolate functional units from each other. By isolating I don't not only mean
   a clear separation of concern I also mean that we can separate them into different processes.

The third point is very important to me since it increases stability tremendously if individual
components are allowed to fail catastrophically without crashing the whole system.

BTW: I am currently working towards getting strider2 (or at least what I think should be strider2) into a state that can actually be used for a simple use case (getting PR notifications from github, transforming these into jobs, executing the jobs on drones and streaming the progress / results to a simple ui). IMHO most of this functionality should be implemented as plugins and following what I wrote previously thus be implemented as microservices.

Concerning your criticism that people wouldn't understand microservices / senca:

• we could write an evenemitter like wrapper around sencea if people really struggle with the concept
  of microservices
• The code I've written is by far not the nicest code I've ever written. I think it could be
  improved in order to increase readability. Thus, I would put the blame for the code not being
  that readable on missing documentation and the need for a refactoring.
  This can be fixed in my opinion by a refactoring of the code and a documentation which describes
  why and how we use seneca.

[phiros]

After using seneca for a while I have to admit that it might get a little bit tricky
to refactor the code so that others can understand it easily.
Therefore, I decided I will refactor my previous code and replace seneca essentially with
EventEmitter2 (signaling / messaging between core components) and waterline (persistence layer).

Since, I have to refactor quite a bit of code: are there any objections against using at least some ES6 (harmony) features?
A few of the new features look really nice to me (block-scoped variables, constants, classes and string interpolation, etc). AFAIK most (non-ancient) node versions support the --harmony flag by now.

[knownasilya]

I think we can assume usage of iojs (no flags required for most stable es6 features) with this version, which will be the next major version of node anyway.

I'm not completely sold on waterline yet, especially since the main contributors aren't so active lately and it's not 1.0 yet. I'd rather continue with the format we have now:
https://github.com/Strider-CD/core/blob/master/lib/models/drone.js so in the future we can just swap out for waterline or whatever if need be.

[knownasilya]

@phiros I'd like development to be more open, so if you are working on something start the PR early, even before you have anything changed yet.

[phiros]

Ok will do