Event Driven Framework

[plauche]

We are considering building an event driven layer on top of the Kubos flight software framework. An initial prototype implementation was created as part of #168 and demonstrates what an event driven interface could look like in our context. However further discussion is needed to settle on some broader architecture directions and implementation details.

Items that still need discussions and/options weighed are:

• Architecture
  • SOA/Brokered - One dedicated message broker stands in-between all software nodes
  • Brokerless - Every node has the ability to act as a broker, but none are dedicated
• Queues
  • What are the potential queue options?
  • Should we be concerned with queue persistence?
• Messaging Layer
  • What are the potential messaging layer options?
    • ZeroMQ http://zeromq.org/
    • Nanomsg http://nanomsg.org/
    • MQTT http://mqtt.org/
• Testing - What will this look like?

[plauche]

Interesting article from ZeroMQ about different messaging architectures http://zeromq.org/whitepapers:brokerless.

[plauche]

Document Event Architecture Discussions

[plauche]

Here are a couple of interesting blog posts about testing message based systems

• https://jeremydmiller.com/2016/05/16/automated-testing-of-message-based-systems/
• https://jeremydmiller.com/2016/05/17/reliable-and-debuggable-automated-testing-of-message-based-systems-in-a-crazy-async-world/

Designing with and around integration tests should be highly prioritized when building out this system.

[plauche]

Some of the differences between ZeroMQ and nanomsg - http://nanomsg.org/documentation-zeromq.html

[OhMeadhbh]

After review of issues above and additional internal discussions w/ Dave, we want to move forward w/ using 0MQ because:

• It has a flexible socket library. we get request / response & pub / sub from the same framework
• We don't have to re-invent the wheel
• It automagically reconnects after process restart

Nanomsg is interesting, but 0MQ seems more mature with a larger community & better support. But who knows, this may change in the future.

Things we sort of wish it did out of the box include:

• Message Persistence
• Guaranteed Delivery
• Explicit "deliver exactly once" vs "deliver at least once" semantics
• Fully supported message broker from the same people who wrote the 0MQ library
  But we're convinced we can move forward without them.

We thought about whether to use a broker or a non-brokered architecture:
Arguments in favour of a non-brokered architecture:

• Single Point of failure
• Potentially faster (you don't have to copy the message from sender to broker and then copy it from the broker to the receiver.)

Arguments in favour of a brokered architecture:

• Much easier to implement message persistence
• We think it would be easier to debug (one process to debug or log messages from instead of multiple.)
• It's probably easier to implement message priorities in a broker than in a non-broker/peer architecture.
• total number of connections w/ brokerless system increases order n! but increases order n with the broker.

Additional Thoughts:
While running through use cases on brokered vs. non-brokered messaging facilities, we realized there were two different types of messages: commands & events (that tell individual systems what to do (like "turn on radio" or "start taking pictures") ) and Jobs (which are much "higher level" like "take a picture over this WGS-82 location while pointed in a particular orientation") The two queues are distinct.

Also, if we're worried about single point of failure (and we are), then we should spend more time hardening the broker instead of debugging weird side-cases from the peer-to-peer / brokerless architecture.

There are probably several situations where we would need to provide responses to pub / sub messages. We think this can be done with unicast messaging.

Next Steps:
Build a prototype "control plane" implemented as a message bus using 0MQ.