Provide more detailed documentation and check missing features

[zamazan4ik]

Hi!

At first, thank you for the library. I am a quite new in Rust and didn't find any other good Rust library for working with actors. So thank you for your job!

I came from C++ world, where I have some experience with another actor framework for C++ - SObjectizer. Since I am mostly familiar with SObjectizer - I am trying to map entities from SObjectizer to entities from bastion. Also I can compare a little bit other stuff like documentation, examples, etc.. So it can be helpful for you to understand, in which places you can improve some things in bastion and make it even better.

Also will be fine i you convert my comparison into some public-available roadmap, in which me and other developers can check current state of implementing different features.

Why it's important: I think I am not the latest developer, who is trying to find an alternative for SObjectizer in Rust world. Also I am sure, that in SObjectizer you can find a lot of useful for end-users features, which would be awesome to have in bastion, since SObjectizer has a long history of development (more than 15 years) and SObjectizer's authors implemented a lot of useful features into the their library. As a SObjectizer user, I can say, that it's awesome library.

I've already asked some questions in Discord channel and got some answers from @vertexclique but I am sure that it should be saved here, on Github, and other users can read this discussion later.

After reading bastion documentation and playing a little bit with examples, I have the following questions/suggestions, which would be nice to be answered/implemented:

• Please add a vocabulary, which is used in your library. What is supervisor, children, how they are mapped to Actor model. For now it's a little bit unclear at least for me, e.g., what the thing supervisor is. You did a great job with documenting classes itself - that's fine. But I am trying to find documentation like it's done in SObjectizer: https://github.com/Stiffstream/sobjectizer/wiki/SO-5.7-Tutorials-Basics
  Please check SObjectizer's Wiki pages - that's an awesome example of a good documentation for end-users. Check not only basic parts - there is a seria "In-Depth" articles, which helps a lot with understanding several parts of SObjectizer.
• Add more examples. From my experience, for almost every feature SObjectizer has an example. You can check them here: https://github.com/Stiffstream/sobjectizer/tree/master/dev/sample/so_5
  Would be awesome if something similar will be available for bastion
• Add some notes about "mind behind bastion". It will help with understanding some design flaws.
• Add documentation, examples (probably some benchmarks) for lightproc and other bastion components since they are important parts of bastion as far as I see.

Let's talk about features, which are missing or I missed in bastion and which I use in SObjectizer. Possibly they are available in bastion now, but I didn't find them, so (by the way, that's a good point for documenting it somewhere :) )

• What is an agent or actor in bastion? How can I create group of actors (they called cooperation in SObjectizer)
• What is an alternative to SObjectizer's dispatcher in bastion? Is it possible to write own dispatcher? Are there any examples? Which types of dispatchers are already implemented in bastion?
• Can an actor be implemented as finite-state machine (with all features, which are available for FSM (history, on_entry/on_exit callbacks, nested states, etc)? Can an actor handle different message types in different states (so if message type isn't supported in current state - just handle it with dead-letter handler)? Is dead-letter handler supported?
• Is it supported any kind of runtime monitoring of whole system in bastion? I am looking for something like this one.
• Which kind of delivery tracing is supported in bastion? I am looking for something like this one.
• Are mutable messages supported?
• Is any mechanism for defining limits for every actor? E.g. SObjectizer has the following mechanism.
• Do actors support priorities? It can be helpful when you run a lot of actors on some limited count of threads, and some actors have higher priority.
• What is an alternative to message box (or mailbox) from SObjectizer? Mailbox is an entity, where message can be sent. Actors can send and receive message to/from mailbox. Every agent has a default mailbox and also can create additional mailboxes (e.g. for separating logic). All mailboxes are registered in global environment (It's called SOEnv). Is there any alternative for it in bastion? Which kind of mailboxes are available? MPSC, MPMC?
• Is there any support for enveloped messages? Are mutable messages supported?
• How actor can interact with outer world? SObjectizer provides message chain mechanism
• What is going on, if in one actor panic is triggered? Can this behaviour be customized?

I am sure, that are not all my questions :) Since I am very interested in your library - I will have them more. Don't think about all my questions as I am trying to show you lack of features or whatever else in bastion - no at all! I am just trying to understand your library and find features, with which I am already familiar and which are important for me.

Thank you!

[o0Ignition0o]

Hi,

Thanks a lot for such a thorough issue. There's a lot of super valuable feedback!

Ill have a thorough look at SObjectizer, but I really like the terms you use in the issue, so it might fit my mental model pretty well!

In the meantime maybe other team members will be able to give you some answers, especially with respect to the Dispatcher that should become a trait at some point, or the Routing methods that allow you to target a child or a dispatcher that will pass the message to one of the children.

Thanks again for taking the time to write this, it's really insightful and very interesting!

[zamazan4ik]

@o0Ignition0o @vertexclique can you provide at least some answers to my questions? I think we can start here collecting answers for all questions and committing them as a documentation to bastion. I think @vertexclique can give answers for a lot of my questions :)

[o0Ignition0o]

can you provide at least some answers to my questions?

Oh sure sorry.

I'm not sure I can reply to all of the questions, so I'll leave it to @vertexclique, but here's a starter:

Add documentation, examples (probably some benchmarks)

We're trying super hard to do so, there's a showcase and a couple of examples right now, but you're more than correct, we need to provide more (and we're trying to ^^)

On top of that, part two of our blog series on bastion and tide seems to come right on point, it starts explaining how to build a bastion from the ground up. I really hope it can be helpful :)

@oScape has been doing quite a bit of effort on it lately as well, #192 and is laying the groundwork for our book.

How actor can interact with outer world?

The outer world can interact with the children through the tell_anonymously and ask_anonymously variants. But doing it the other way around might involve sending a channel::sender() to the children when they spawn, or something else I haven't thought of.

an an actor be implemented as finite-state machine (with all features, which are available for FSM (history, on_entry/on_exit callbacks, nested states, etc)?

IIRC, @Relrin started working on Children and Supervisor callbacks that might come in handy to implement lifecycle related logic. It's probably not as deep as SObjectizer, but maybe that's what you're looking for?

Do actors support priorities?

I don't think they do

What is an alternative to message box (or mailbox) from SObjectizer? Mailbox is an entity, where message can be sent.

Mailboxes exist in Bastion, and they're useful for us to implement mailbox preservation when a child gets restarted / children get scaled, but I'm not sure they are the same as SObjectizer's though. I need to dig deeper into SObjectizer so I can be accurate enough with my answers.

The last part is only my opinion for now, so it doesn't reflect the teams direction or anything, I'm just trying to poke around that and hope something cool will come out of it:

I'm trying to work towards removing the msg! macro as shown here: #214 , which will probably have a lot to do with some TypeLevel programming / Associated types.

With that in mind send/ask might move to traits, and enable "actors" impl by defining impl Ask / Send for your struct.

I'm not really sure how dispatchers work in SObjectizer, but I feel like a Dispatcher trait would allow you to perform something similar.

Long story short I would like to try to define a clean Supervisor trait, a Dispatcher trait, and a Child trait and have them work together so anyone can define a custom behavior, but I'm not too aware of the deep inner workings of bastion yet, so my opinion might change in the future.

I would however love to be able to discuss the approach I'm willing to take with you, so you can tell me more about your needs and if our design decisions would fit them!

Hope this addresses some of your questions/concerns :)

[zamazan4ik]

@o0Ignition0o Thank you for the answers! They are really helpful. Will be awesome to see answers for other questions :)

[zamazan4ik]

@vertexclique can you please provide more details according to my questions? Thank you!

[vertexclique]

Bear with me. I need some time, and I will respond to this issue when things in my plate emptied. I didn't forget what you've written.

[zamazan4ik]

@vertexclique maybe you just forgot about the issue :) Just a friendly reminder :)

[resolritter]

I'll answer from my experience from using the library, but bear in mind I'm no core developer so I won't answer everything and the details might be incomplete.

First of all, regarding your plea for "more examples", you should first assess what is missing from the examples. Asking for more examples without telling what you want is the same as asking for someone else to do two things: (1) the job of figuring out what is missing from the examples and (2) writing the examples; both is going too far IMO. It's also worth noting that, when a PR adds code which expands the API, it usually already includes an example.

What is an agent or actor in bastion?

An agent's code runs from a closure (example) which is put in a Children (example). To my knowledge, those closures are wrapped with LightProc (source). Going through the call chain:

bastion/src/bastion/src/child.rs

Line 386 in aa066f2

pool::spawn(self.run(), stack)

bastion/src/bastion-executor/src/pool.rs

Line 47 in aa066f2

pub fn spawn<F, T>(future: F, stack: ProcStack) -> RecoverableHandle<T>

bastion/src/bastion/src/child.rs

Line 365 in aa066f2

match poll!(&mut self.exec) {

How can I create group of actors

By specifying a redundancy > 1 (example), N actors will be spawned in the same group

Can an actor be implemented as finite-state machine

Posting an example of what you want to do will make answering this easier. That being said, if I were to implement a state machine, it could start like this (pseudo code):

    children.with_exec(
        move |ctx: BastionContext| async move {
            // initialize the state
            let mut state = "START";

            loop {
               let msg = ctx.recv().await?;
               match msg {
                   State::Continue => state = "RUN",
                   State::Reset => state = "START"
               }
            }

            Ok(())
        }
    )

Since loop will run infinitely and ctx.recv().await? will "block" until the next message arrives in the inbox, state can transition as much as you want. Also, since the actor might panic and the state would be lost in that case, instead you could initialize the state outside of closure and just read it again when the actor's closure is spawned again.

Can an actor handle different message types in different states

Yes, you just have to wrap the different messages in an enum. That is what I did and explained here.

What is an alternative to SObjectizer's dispatcher in bastion

See Dispatcher

Is it supported any kind of runtime monitoring of whole system

You could use tracing and logging crates within the actor's closure or callbacks to record whatever information you want throughout an actor's lifecycle. See the callbacks example.

On my project I've used a macro called logged (see here) to automatically log all messages received in all actors, then I saved the log as snapshots (example) during tests runs.

How actor can interact with outer world

You can spawn an actor dedicated to getting inputs (example) from the outside world. For example, you could spawn a web server, or connection to a message queue, or a socket, or an IPC channel, or whatever else you want for I/O, parse the input, pack the message, then use the BastionContext for sending the message across the system.

On my project I'm hosting a web server inside of an actor (source), although I'm not using the BastionContext to send messages due to #287; instead, crossbeam channels are used (source).

What is going on, if in one actor panic is triggered? Can this behaviour be customized?

Since the children's execution is wrapped with LightProc, a panic will trigger sending message to the supervisor, which will, in turn, decide what to do based on the restart strategy. Searching for "message restart" will lead you there.

https://github.com/bastion-rs/bastion/search?q=message+restart

bastion/src/bastion/src/supervisor.rs

Line 819 in 1383c19

async fn restart(&mut self, objects: Vec<RestartedElement>) {

To my knowledge, it's customizable in the sense that you could write your own runtime leveraging the same crates as the Bastion runtime does, although I personally don't currently know how to really pick it apart.

[zamazan4ik]

@resolritter Thank you a lot for such detailed feedback! My bad that I've seen it only now...