[WIP]Allow force deleting an actor from any state.

[Sneha-at (Sneha-at)]

Fixes #643

It's a good idea to open an issue first for discussion.

The "Force Delete" feature allows deleting an actor from almost any state (e.g., RUNNING, CRASHED, SUSPENDING), bypassing the normal restriction that required the actor to be SUSPENDED before deletion. This is crucial for cleaning up stuck or failed actors.

Involved Components

• Ateapi (Control Plane): Orchestrates the deletion workflow. It manages the actor state machine, communicates with Atelet, and updates the global store.
• Store (Database): Holds the state of Actors, Workers, and Volumes.
• Atelet (Node Agent): Runs on each worker node. It receives termination requests from Ateapi, delegates to Ateom, and performs node-level cleanups (unmounting volumes, clearing directories).
• Ateom (Runtime Agent): Runs on the worker node, managing the actual workload execution environment (gVisor sandbox or MicroVM). It handles the low-level termination of the workload process and network cleanup.

sequenceDiagram
    autonumber
    actor Client
    participant Ateapi as Ateapi (Control Plane)
    participant Store as Store (DB)
    participant Atelet as Atelet (Node Agent)
    participant Ateom as Ateom (Runtime Agent)
    participant VM_Sandbox as VM/Sandbox (gVisor/CH)

    Client->>Ateapi: DeleteActor(Actor, force=true)
    activate Ateapi
    Ateapi->>Store: GetActor(Actor)
    Store-->>Ateapi: Actor State
    
    Note over Ateapi: MarkTerminatingStep
    Ateapi->>Ateapi: Verify state (allows almost any state if force=true)
    Ateapi->>Store: UpdateActor(Status=TERMINATING)
    
    alt Actor has Worker Assignment
        Note over Ateapi: CallAteletTerminateStep
        Ateapi->>Atelet: Terminate(ActorUID, TargetAteomUID)
        activate Atelet
        
        Atelet->>Ateom: TerminateWorkload(ActorUID)
        activate Ateom
        Ateom->>Ateom: Deactivate Networking
        
        alt Runtime is gVisor (ateom-gvisor)
            Ateom->>VM_Sandbox: runsc delete (containers)
            Note over Ateom, VM_Sandbox: Kills and deletes sandboxed containers
        else Runtime is MicroVM (ateom-microvm)
            Ateom->>VM_Sandbox: Shutdown VMM / Kill Process
            Note over Ateom, VM_Sandbox: Shuts down Cloud Hypervisor & virtiofsd
        end
        
        Ateom->>Ateom: Unmount OCI Overlays
        Ateom->>Ateom: Cleanup Actor Network
        Ateom-->>Atelet: TerminateWorkload Response
        deactivate Ateom
        
        Atelet->>Atelet: Unmount External Volumes on Node
        Atelet->>Atelet: Reset Actor Directories on Node
        Atelet-->>Ateapi: Terminate Response
        deactivate Atelet
    end

    Note over Ateapi: DetachVolumesForDeleteStep
    Ateapi->>Ateapi: Detach volumes from actor in control plane

    Note over Ateapi: ReleaseWorkerStep
    Ateapi->>Store: Get Worker
    Ateapi->>Store: UpdateWorker(Assignment=nil) (releases worker)
    Ateapi->>Store: UpdateActor(WorkerAssignment=nil, LocalSnapshotInfo=nil)

    Note over Ateapi: MarkDeletingStep
    Ateapi->>Store: UpdateActor(Status=DELETING, Volumes=DELETING)

    Note over Ateapi: DeleteVolumesStep
    Ateapi->>Ateapi: Delete volumes (durable storage)

    Note over Ateapi: FinalizeDeletedStep
    Ateapi->>Store: DeleteActor() (removes record)
    
    Ateapi-->>Client: Deleted Actor

Loading

Detailed Step Descriptions (Control Plane)

1. LoadActorForDeleteStep: Fetches the latest actor state and template from the store.
2. MarkTerminatingStep: Transition the actor status to TERMINATING. If Force is true, this transition is allowed from any active state.
3. CallAteletTerminateStep: If the actor is currently assigned to a worker node, this step dials the Atelet on that node and requests termination.
4. DetachVolumesForDeleteStep: Initiates detachment of any volumes associated with the actor.
5. ReleaseWorkerStep: Clears the assignment on the worker resource in the store, making the worker available for other actors. It also clears the worker assignment on the actor record.
6. MarkDeletingStep: Transitions the actor status to DELETING and marks its volumes as DELETING.
7. DeleteVolumesStep: Deletes the actual volume resources (e.g., GCS buckets or directories).
8. FinalizeDeletedStep: Permanently removes the actor record from the store.

• Tests pass
• Appropriate changes to documentation are included in the PR