Related issue: #641
Once an Actor is in RESUMING status, the only allowed transition paths are:
- Success scenario: Actor becomes RUNNING
- Crash scenario: Actor becomes CRASHED
That means if some error happened during Resume, the only way out is for users to keep on retrying until it succeeds, or abandon and leak the actor.
#119 proposes a "revert" API, which could help with restoring to the last known state, but even that could also run into errors.
I think as a final escape hatch, we should provide some "force delete" option. where we can 1) trigger (perhaps periodically) a cleanup sequence on nodes and CP 2) still keep track of that actor and potentially leaked resources, but allow for new actors with the same name to be created.
To accomplish this, we probably have to:
- Track force deleting actors under a different key, maybe we prefix the actor UID with some keyword like "forcedeleting"
- Have periodic garbage collectors in both CP and atelet.
- Atelet only GCs actor UIDs that have been marked as forcedeleting.
- CP GC waits for atelet GC for that actor UID to complete. This implies that we need to keep track of which atelet the actor was running on and also that we wouldn't have allowed the actor to leak on multiple nodes.
Any other thoughts/ideas?
Related issue: #641
Once an Actor is in RESUMING status, the only allowed transition paths are:
That means if some error happened during Resume, the only way out is for users to keep on retrying until it succeeds, or abandon and leak the actor.
#119 proposes a "revert" API, which could help with restoring to the last known state, but even that could also run into errors.
I think as a final escape hatch, we should provide some "force delete" option. where we can 1) trigger (perhaps periodically) a cleanup sequence on nodes and CP 2) still keep track of that actor and potentially leaked resources, but allow for new actors with the same name to be created.
To accomplish this, we probably have to:
Any other thoughts/ideas?