High Concurrent Active Actors per Worker

[brandonrjacobs]

Reading the architecture and overview, as I understand it only one actor per worker can be running / active at once. Is the goal to support multiple active actors per worker in the future?

[thockin]

We spent some time on this topic in the past. My fairly strong feeling (but always convinceable) is that it's WAY more complicated to support 1:N than 1:1. Simplicity is really important here, especially in the first 6 months -- we have a LOT of work to tackle, this seems like a distraction.

1 worker to N actors means:

• intra-pod performance isolation (e.g. cgroups)
• more complicated pod lifecycle (externally e.g. termination, draining)
• more complicated intra-pod lifecycle management (approximating a full kubelet)
• more complicated autoscaling
• more complicated resource management (e.g. k8s in-place updates)
• more complex scheduling
• more complex APIs and CLIs

So I would flip it around - why would we want to do 1:N instead of 1:1 ?

[bittermandel]

I wasn't aware this was the case until I read this issue. Just so I understand; does this mean one worker per node? It doesn't feel like it would take advantage of running really big nodes to scale better horizontally.

For background: our agent is centralized outside of the sandbox (or actor in this case) and requires a long-running running connection, which would be tricky to achieve here. On top of that, our users interact with the sandboxis through a Vite HMR connection directly to the process. This experience would be jittery, prone to failure and require a proper control plane + decent resumption support across our application if we need to cycle through actors at all times.

In terms of resources, at our scale I believe we'd spend significant resources suspending and resuming workloads, though definitely less than it would take to just keep them running. Probably that'd be many gbit/s across all nodes for the snapshotting & resuming, which also isn't free.
Could we run one worker per core perhaps with session pinning and disk caching of snapshots etc? There's more ideas but I'm struggling to see how to scale this type of infra.

[BenTheElder]

wasn't aware this was the case until I read this issue. Just so I understand; does this mean one worker per node? It doesn't feel like it would take advantage of running really big nodes to scale better horizontally

No. One per pod, which runs an interior sandbox that mutates to run different actors as needed. But each of those "ateom" pods runs one workload at a time. When the workload is suspended the same pod can host another workload.

A node has one "atelet" to coordinate amongst the "ateom" pods hosting workloads, which are many.

It already supports multiple workers per node this way, as you can see in the demo in the README.

[thockin]

Jonathan Grahl (@bittermandel)

I should start with acknowledging that there is a need to balance concerns here. We want to keep the infrastructure as simple as possible but also want to impose as little as we can upon applications while delivering efficiencies.

The current working model is that a given kube cluster can run as many workers as needed - they are "regular" pods to kubernetes, with resource requests etc. Those workers can become any actor, on the fly, via resume-from-snapshot. They can execute as long as they need to and then suspend.

If an agent never suspends, it is difficult to get the efficiency.

The interesting part, IMO, is that any worker can become any actor (modulo worker class which is not introduced yet). This breaks the hard-association to a single machine, making everything more fungible, at the cost of network, as you called out.

Then we have to deal with reality.

The demo state saves EVERY snapshot to network storage, which will easily overload the network. Even if we get more incremental with snapshots, it's a lot.

So we're working backwards from there -- can we save state locally "for a while" (#119) and save to storage only when the higher-order system knows it is worthwhile? That presumes "local" media exists and isn't just a veneer over the same network.

There's still a cost to that - suspended actors are serialized and need to be resumed, which is not "free" but is far less expensive than squirting them across the network.

This experience would be jittery, prone to failure and require a proper control plane + decent resumption support across our application if we need to cycle through actors at all times.

Can you say more about this? Assume for argument that:

• 99% of activations which occur within 5min of the previous suspend will happen in 200ms, the remaining 1% happen in 500ms.
• If you resume a suspended actor after 5 min, that changes to 50/50.
• After suspend/resume, the actor can comes up with a different IP address, so network connections should not be assumed to survive.

All those numbers as re made up for now, but imagine the SLO built around those ideas. An important part of that is that YOU decide when to suspend. If you think it is worth waiting for 30s because that's 99% of your users' turn-to-turn latency, then you avoid a lot of the above, at the cost of consuming that worker for 30sec.

An alternative model could be that some high percentage of "frequent" activations come back on the same worker (with the same IP) but there's a wider variance in resume time (e.g. 90p might be 200ms but 99p is 3 seconds). In other words, are you willing to wait in order to get the same worker / IP?

Another option might be to demand IP mobility (IPv6) such that every actor has an IP that travels with it. I don't think many kube implementations can do that, but it should be on the table.

Another option might be to embrace this proposal in a BIG way - create a single mega-worker per node which is responsible for managing many actors internally. This is basically a node within a node. Kubernetes manages the machines, but only sees a single "physical" pod. We could better leverage things like OS swap support, at the cost of much more complicated interior management. It would need to track lots of IPs to dole those out to actors, etc. Basically creating a parallel Kubelet+CRI+CNI stack.

All options should be on the table.