Design discussion: WorkerPool vs. node-level Actor runtime

[WenYuLuo]

Context

A WorkerPool is currently reconciled into a Deployment of Worker Pods. Each
Worker has a single Actor assignment, and the scheduler selects an available
Worker:

• https://github.com/agent-substrate/substrate/blob/main/cmd/atecontroller/internal/controllers/workerpool_apply.go#L151-L167
• https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/internal/scheduling/scheduling.go#L89-L125

Actor-level resource requests are not currently part of ActorTemplate or
WorkloadSpec; resource configuration is attached to the Worker Pod:

• https://github.com/agent-substrate/substrate/blob/main/pkg/api/v1alpha1/workerpool_types.go#L22-L52
• https://github.com/agent-substrate/substrate/blob/main/pkg/api/v1alpha1/actortemplate_types.go#L97-L153

The current networking/runtime implementation also assumes at most one active
Actor per Worker.

Design question

What concrete requirement motivates using a Worker Pod as the capacity unit,
rather than a node-level DaemonSet/agent that manages multiple per-Actor gVisor
or microVM sandboxes?

The sandbox appears to provide the primary Actor isolation boundary. Node
capabilities such as /dev/kvm, mounts, and networking could also be exposed
to a node-level runtime manager. The current WorkerPool model may introduce
fixed-size resource bins and additional Pod/runtime overhead, while a node
failure still affects all Worker Pods on that node.

Discussion points

• Is WorkerPool intended to represent warm concurrency slots, resource
  classes, or a security/failure domain?
• How should heterogeneous Actor CPU/memory requirements be modeled and
  scheduled?
• What Pod-specific property is required beyond the sandbox and runtime
  manager?
• Could WorkerPool represent a runtime/resource class while a node-level
  agent schedules multiple Actor sandboxes?

I may be missing constraints around security, snapshot locality, or runtime
lifecycle. The goal of this issue is to clarify the intended abstraction and
trade-offs, rather than propose an immediate implementation change.

[BenTheElder]

What concrete requirement motivates using a Worker Pod as the capacity unit,
rather than a node-level DaemonSet/agent that manages multiple per-Actor gVisor
or microVM sandboxes?

This is being considered, but it's a massively breaking change and there are other priority fixes to land.

Time to market. The worker pod model had some nice scheduling properties to ship something. It's not clear this will land longterm.

cc Dmitry Berkovich (@dberkov) Tim Hockin (@thockin)

[BenTheElder]

If we pivot to node level, I also don't think we should make it only support one or the other. As long as /dev/kvm is available we can have the same CPU/memory slice support either [gvisor, uvm] under the node-level daemon.

[WenYuLuo]

Thanks for the clarification, Benjamin Elder (@BenTheElder). It is helpful to understand that the Worker Pod model was primarily chosen for time to market and that a node-level design is still being considered.

Two follow-up questions:

1. Could you elaborate on the specific scheduling properties that made the Worker Pod model attractive? For example, are the main benefits Kubernetes resource placement, warm-capacity management, snapshot locality, failure handling, or something else? Understanding this would help make the trade-offs against a node-level agent more concrete.

2. Are there concrete roadmap milestones or conditions for moving toward the node-level design? For example, an Actor-level resource model, a node-agent prototype, density/fragmentation targets, or a particular workload benchmark. It would be useful to know whether this is expected to remain a future exploration or is planned after specific priorities are completed.

The note that a node-level daemon could support both gVisor and VM runtimes using per-Actor CPU/memory slices is also helpful. Thanks again for taking the time to clarify the direction.