Memory leak in a long-running program that spawns actors

[xquelard]

Hi there! I also shared this question on spectrum (see https://spectrum.chat/statecharts/general/memory-leak-in-xstate-when-spawning-actors~edb45f68-c6d9-4cc0-b2fa-d1d73f314e0d ),but I thought I would be useful for xstate users to see my question asked here.

So, here is the problem I'm facing :

We have a component in production that orchestrates some operations on our infrastructure, and we used XState for that component. Recently, we noticed that it was rebooting regularly because of a memory leak, so I'd like to understand where it comes from.
We have a top-level interpreter that is basically just a bonified container, as it stores in its context a dictionary of actors. When certain transition are effectued, we spawn a new actor, and register it in the context of the container, for later use.
I noticed in that component that, when we needed to remove an actor from the container, we did NOT use the stop method on it, we simply removed it from the container's dictionary. And indeed, after some tests, I confirmed that this is not enough to get rid of the actor, as it is still registered in the container property called 'children'. So, as of now, by best theory is that, as more and more actor were being spawned into memory, we eventually ran out of it.
However, even when I call stop() on the actor, the actor is still referenced in the topLevelContainer.children. You can see this in action on this REPL : https://repl.it/@xababafr/XstateMemoryLeakPoC2#index.js
On the second run of this REPL, I expected the 'actor2' to not be saved anymore in the actorsContainer.children.

So my question is : if stopping the actor is not enough to free the memory, what is? I'm not sure what's necessary in order to make some room for new and active actors

[Andarist]

Please see #1434 . It most likely answers your questions.

[xquelard]

Thank you for your response. As discussed below, I'll probably just clean up myself for now, and remove that code when the issue is solved in a later release !

[davidkpiano]

Yeah, I think this may be an issue on our side - calling someActor.stop() currently does not notify the parent that the actor should be removed (and GC'd). I'll look into this. Do you want to make this an issue for tracking?

[xquelard]

Sure, I will create an issue for tracking purposes :) ! I the mean time, it there anything I can do on my side to fix the problem? I've looked at the public API and I haven't seen anything that could clean up the parent, but I may have missed something (or maybe I can even just remove the actor from the interpreter by hand? But i'd be pretty scared of side effects If I were to go this way)

[davidkpiano]

or maybe I can even just remove the actor from the interpreter by hand?

Yes, but you'd have to delete it from state.children as well as service.children.

[xquelard]

great ! Thanks for the quick replies, for now we'll go for the quick-but-dirty solution :) I've created the issue #1457 as you asked :)

[Andarist]

Prior art: in Akka, a parent actor is notified when a child actor "dies" (or is stopped) via a watcher/supervision (basically the same as sending an event to the parent).

This is very reasonable for me and extra implicit onError handler should probably do something like this. My problem is that in this particular situation we're not talking about a child actor crashing but rather a child actor being stopped forcefully from outside. Question is - should it even be allowed?

In Akka ActorRef is used to communicate with a given actor and it can be shared across the actor tree freely or probably even shared remotely, but it has no "stop" method. By just obtaining an "address" to an actor you can't just stop/kill it - which is right now possible in XState (v5). And this I have a problem with as it seems to give too much power to external actors.

[davidkpiano]

From Akka:

Classic actors can be stopped with the stop method of ActorContext or ActorSystem. In Typed an actor stops itself by returning Behaviors.stopped. There is also a stop method in the ActorContext but it can only be used for stopping direct child actors and not any arbitrary actor.

PoisonPill is not supported in Typed. Instead, if you need to request an actor to stop you should define a message that the actor understands and let it return Behaviors.stopped when receiving that message.

https://doc.akka.io/docs/akka/current/typed/from-classic.html#stopping

We can probably supplant the stop() action in v4 to stop child actors:

on: {
  stopChild: {
    // Actor ID, must be direct child of parent (spawned or invoked)
    actions: stop('some-child')
  }
}