Large image pulls death-loop in `ResumeActor` and amplify atelet RSS

[jjamroga]

Expected Behavior

When ate-controller reconciles a new ActorTemplate, the golden-actor ResumeActor RPC should complete and the template should transition through PhaseResumeGoldenActor → PhaseWaitGoldenActor → PhaseReady. For any image whose cold pull + extract + gVisor restore can reasonably complete, this should happen within one or a few reconcile attempts.

atelet RSS during this path should be bounded by the image's working-set size, regardless of whether individual ResumeActor calls succeed or are retried.

Actual Behavior

For container images whose cold-pull path (registry fetch + decompress + extract + gVisor restore) exceeds ~28 seconds:

1. Death loop on ResumeActor. Every call returns DeadlineExceeded after ~28s and ate-controller's reconciler retries with exponential backoff (~30-60s). The ActorTemplate sticks in PhaseResumeGoldenActor and never reaches PhaseReady. The deadline is hardcoded at cmd/ateapi/internal/controlapi/workflow.go:145 via acquireActorLock(ctx, id, 30*time.Second, 2*time.Second), which couples the Redis lock TTL and the workflow deadline into a single 28s budget.

2. Atelet RSS amplification. Each cancelled pull leaves layer-blob HTTP goroutines on atelet blocked in Body.Read holding partial-blob buffers, because internal/memorypullcache/memorypullcache.go:123-136 doesn't pass the caller's ctx into remote.Image. RSS amplifies proportionally to the loop duration. Observed climb-then-recede shape on a kind node:

   atelet   136m   2453Mi    # baseline
   ...
   atelet   195m   9410Mi    # peak during loop — partial-blob buffers from cancelled pulls
   atelet   588m   6791Mi    # receding as orphaned HTTP goroutines finally drain
   

   Not a permanent leak: orphaned goroutines drain when their TCP connections eventually drop or when a retry's pull happens to complete. But on a production node that's already tight, the spike is enough to OOM-kill atelet — taking every other actor on that worker down with it, not just the one being resumed.

Observed ate-controller output during the loop:

ERROR Reconciler error ... error: "while resuming golden actor: rpc error:
code = DeadlineExceeded desc = context deadline exceeded"

Steps to Reproduce the Problem

1. Set up kind + install substrate:

   ./hack/create-kind-cluster.sh
   ./hack/install-ate-kind.sh --deploy-ate-system

2. Build the ateom-gvisor image into the kind-local registry and capture the resolved reference:

   export KO_DOCKER_REPO=localhost:5001
   export KO_DEFAULTPLATFORMS=linux/$(go env GOARCH)
   ATEOM_IMAGE=$(./hack/run-tool.sh ko build -B ./cmd/ateom-gvisor)
   echo "ateom image: $ATEOM_IMAGE"   # localhost:5001/ateom-gvisor@sha256:...

3. Install metrics-server (needed for kubectl top to verify atelet RSS):

   kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml
   kubectl scale -n kube-system deploy/metrics-server --replicas=1
   kubectl patch -n kube-system deploy metrics-server --type=json \
     -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
   kubectl rollout status -n kube-system deploy/metrics-server

4. Apply the ActorTemplate with the offending image:

   kubectl apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: ate-repro
   ---
   apiVersion: ate.dev/v1alpha1
   kind: WorkerPool
   metadata:
     name: repro-pool
     namespace: ate-repro
   spec:
     replicas: 1
     ateomImage: ${ATEOM_IMAGE}
   ---
   apiVersion: ate.dev/v1alpha1
   kind: ActorTemplate
   metadata:
     name: repro-fat-image
     namespace: ate-repro
   spec:
     workerPoolRef:
       name: repro-pool
       namespace: ate-repro
     runsc:
       amd64:
         url: "gs://gvisor/releases/nightly/2026-05-19/x86_64/runsc"
         sha256Hash: "a397be1abc2420d26bce6c70e6e2ff96c73aaaab929756c56f5e2089ea842b63"
       arm64:
         url: "gs://gvisor/releases/nightly/2026-05-19/aarch64/runsc"
         sha256Hash: "1ba2366ae2efceba166046f51a4104f9261c9cb72c6db8f5b3fe2dc57dea86b9"
     pauseImage: "registry.k8s.io/pause:3.10.2@sha256:f548e0e8e3dc1896ca956272154dde3314e8cc4fde0a57577ee9fa1c63f5baf4"
     containers:
       - name: fat
         image: ghcr.io/kagent-dev/nemoclaw/sandbox-base@sha256:d52bee415dc4c0dba7164f9eabe727574c056d4f211781f20af249707883a3b4
         command: ["/bin/sh", "-c", "sleep 3600"]
     snapshotsConfig:
       location: gs://ate-snapshots/repro/
   EOF

   No manual kubectl ate resume actor needed. ate-controller calls ResumeActor against the golden-actor ID as part of the AT reconcile — that's enough to exercise the bug.

5. Confirm the symptoms:

   # Death loop in ate-controller
   kubectl logs -n ate-system deploy/ate-controller --tail=0 -f | grep -i "deadline\|golden"
   
   # ActorTemplate stuck (expect: ResumeGoldenActor)
   kubectl get actortemplate -n ate-repro repro-fat-image -o jsonpath='{.status.phase}{"\n"}'
   
   # atelet RSS climb-then-recede
   while true; do date; kubectl top pod -n ate-system -l app=atelet --containers; sleep 5; done

If the bug does not trigger on your bandwidth: the cold-pull path completes inside 28s on fast corp LAN / warmed registry mirrors. Force it by throttling the kind node (docker exec kind-control-plane tc qdisc add dev eth0 root tbf rate 10mbit burst 32kbit latency 400ms — apt-get install -y iproute2 first if tc is missing) or by swapping image: for something much larger (e.g. nvcr.io/nvidia/pytorch:24.05-py3 at ~12GB). The death loop sometimes escapes via bandwidth variance + HTTP keep-alive warmup; watch for at least 5 minutes before concluding it didn't reproduce.

Specifications

• Version: HEAD of main
• Platform: kind (single-node, control-plane only), reproduced on darwin/arm64; expected to reproduce on any platform where the cold-pull path for the chosen image exceeds ~28s

[jjamroga]

Following up shortly with a small PR to fix the memory leak.

[jjamroga]

Large image pulls incidentally worked before this issue was fixed, at the cost of the large memory leak. The followup to achieve parity with previous behavior without the memory leak requires addressing the issue below.
#233