DeleteActor never cleans the actor's node directory

[igooch]

Expected Behavior

Deleting an actor releases everything the actor held, including its node-local state: after
DeleteActor, /var/lib/ateom-gvisor/actors/<uid>/ should be removed (or at minimum contain
nothing that marks the actor as still placed on the node).

This matters doubly once #463 Phase 2 lands: the image-cache GC's root set is the per-bundle
rootfs-overlay.json specs — "spec on disk" means "actor placed here; do not evict its image."
A deleted actor should not root an image.

Actual Behavior

DeleteActor is a pure control-plane operation: it deletes the store record and cleans up volumes
(cmd/ateapi/internal/controlapi/delete_actor.go), but nothing ever tells the node. atelet's RPC
surface is only Run/Checkpoint/Restore, and resetActorDirs runs lazily — at the start of the
next Run/Restore or the end of a Checkpoint for the same actorUID. A deleted actor's UID
never comes back, so /var/lib/ateom-gvisor/actors/<uid>/ persists until the node is replaced.

What leaks depends on the state the actor was deleted from (the store allows SUSPENDED or CRASHED,
cmd/ateapi/internal/store/ateredis/ateredis.go):

• Deleted from CRASHED: a crash means no Checkpoint ran after the last Run/Restore, so the
  entire bundle dir survives — the actor's writable upper/ layer and the
  bundles/<container>/rootfs-overlay.json spec. Post-Design: on-disk, layer-deduplicated OCI image cache for atelet #463-Phase-2, that leaked spec is
  indistinguishable from a live one, so the dead actor's image is rooted forever: GC logs it as
  in-use on every pass and it can never be evicted, regardless of disk pressure. Hit twice during
  Phase 2 GC validation. The GC is behaving as designed (ambiguity must resolve toward retention —
  evicting a mounted lowerdir is catastrophic); the defect is that deletion leaves the "I am placed
  here" marker behind. Filed separately from Design: on-disk, layer-deduplicated OCI image cache for atelet #463 so this over-retention isn't mistaken for a GC bug.
• Deleted after a clean suspend (the common path): the bundle was wiped at Checkpoint, but the
  ActorPath skeleton, sandbox-assets.json, and any local-checkpoint/ contents are left
  behind.

Steps to Reproduce the Problem

1. On a Kind cluster with substrate installed, run an actor, then crash it (e.g. kill its
   workload) so it lands in CRASHED.
2. kubectl ate delete actor <name> — succeeds (CRASHED satisfies the delete precondition).
3. On the node: docker exec <kind-node> ls /var/lib/ateom-gvisor/actors/<uid>/bundles/<container>/
   → rootfs-overlay.json, upper/, work/, rootfs/ still present, forever.
4. (Variant for the suspended path) Suspend an actor, delete it, and observe local-checkpoint/
   and sandbox-assets.json survive under the actor's dir.
5. (Post-Design: on-disk, layer-deduplicated OCI image cache for atelet #463-Phase-2) Put the node under disk pressure: GC logs the deleted actor's image as
   rooted on every pass; it is never evicted.

Anything else we need to know?:

The fix belongs in the deletion path, not in GC: something should remove ActorPath(uid) when the
actor is deleted. Cleanup should key on deletion (the actor record no longer exists), not on
the CRASHED transition — whether a crashed actor should be recoverable from its node-local state
is an open design question on #292, and deletion-keyed cleanup is safe under either answer. Two
candidate approaches:

• Control plane calls atelet at delete time (a new teardown RPC). Direct, but the control plane
  must still know — and be able to reach — the actor's last node when the delete happens.
• atelet reconciles ActorsDir against the control plane (periodic sweep removing dirs for
  actors that no longer exist). Self-healing — it also catches missed deletes and actors crashed on
  dead workers ([P3] releaseActorOnDeadWorker does not clear AteomPodUid — diverges from crashActor cleanup #618) — but it gives atelet a control-plane client it deliberately doesn't have
  today.

A full "delete all" of the directory is safe: nothing can reference a deleted actor's node-local
state, because the actor record that would point to it is gone.

Related: #615 (node-side debris on an abnormal path), #172 (local snapshot storage management
umbrella).

Specifications

• Version: main @ e1fa004 (2026-07-28); GC interaction observed with the Design: on-disk, layer-deduplicated OCI image cache for atelet #463 Phase 2 POC branch
  (phase2-gc-record-first @ a06ee86)
• Platform: Kind (v1.36.1, Debian GNU/Linux 13 (trixie), 6.8.0-117-generic (arm64),
  containerd://2.3.1) and GKE (v1.35.5-gke.1163012, c3-standard-4, 6.12.85+, containerd://2.1.7) —
  behavior is identical on both