ateapi/controlapi: retry worker updates on version conflict in syncer - #578
ateapi/controlapi: retry worker updates on version conflict in syncer#578Troy Chiu (troychiu) wants to merge 1 commit into
Conversation
| slog.InfoContext(ctx, "Syncer: updating worker in store (labels changed)", slog.String("worker", pod.Namespace+"/"+pod.Name)) | ||
| w.Labels = pool.GetLabels() | ||
| changed = true | ||
| // TODO: We perform an inline retry loop here using wait.ExponentialBackoff |
There was a problem hiding this comment.
Should we just make that change now? How large would it be? IIRc The ActorTemplate controller already uses requeue
There was a problem hiding this comment.
Yes ActorTemplateController is built using controller-runtime, which automatically provides a workqueue and handles retries when Reconcile() returns an error or RequeueAfter.
The refactor appears moderate, involving approximately 80–100 lines of changes in syncer.go to implement a PodLister, workqueue lifecycle, and worker goroutine loop.
We could keep this PR focused on the small inline backoff fix to handle version conflicts immediately, and I can create a separate PR for the refactor.
What do you think?
There was a problem hiding this comment.
Given that most of this will be replaced by that change I’d vote to just close this in favor of that, or just change this PR
There was a problem hiding this comment.
Sounds good. I can repurpose this PR. Thanks for the feedback.
|
Closed this PR since we are going for #674 |
2 participants
This PR adds retry logic with exponential backoff when
WorkerPoolSyncerencounters optimistic locking conflicts (store.ErrVersionConflict) while updating worker state in the store.Before:
syncWorkerToStoreattempted a singleUpdateWorkerwrite. If a concurrent write modified the worker in Redis, it failed withstore.ErrVersionConflict, logged an error, and silently dropped the metadata update (SandboxClass,Labels).Now: On
store.ErrVersionConflict,WorkerPoolSyncerretries up to 5 times with exponential backoff, re-fetching the latest worker state from Redis on each attempt before re-applying changes.Tests pass
Appropriate changes to documentation are included in the PR