Identify what metrics we should use for HPA-based workerpool autoscaling

[laoj2]

Today, the HPA demo is configured to autoscale a workerpool based on ate.workerpool.workers metric:

substrate/demos/autoscaled-workerpool/prometheus-adapter.yaml

Lines 47 to 57 in e70e1f2

	config.yaml: |
	externalRules:
	- seriesQuery: '{__name__="ate_workerpool_workers"}'
	resources:
	namespaced: false
	name:
	matches: "^(.*)$"
	as: "$1"
	# max() across ate-api-server replicas since each replica reports the
	# global count.
	metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>}) by (ate_workerpool_namespace, ate_workerpool_name, ate_worker_state)'

ate.workerpool.workers counts workers by state, so it's clamped by capacity: eg, when a pool of 5 is fully assigned it reports 5, whether one actor is waiting or a thousand are.

Scaling on this metric alone may be fine for the steady state, but we'll be blind to incoming demand that greatly exceeds the pool capacity.

atenet.router.parking.active is a demand sign, but it doesn't know what specific workerpool the demand is for . A control plane equivalent metric doesn't exist.

One option is to autoscale on a new ate.workerpool.demand (filtered on pool_exhausted blocked reason) metric that counts actors that want a slot:

attribute	values
ate.workerpool.name	pool name; empty when nothing in the fleet was eligible
ate.sandbox.class	gvisor / microvm
ate.demand.state	resident (holding a slot now) / blocked (rejected/still retrying)
ate.blocked.reason	none, pool_exhausted, node_locality, no_eligible_pool, unknown

[shrutiyam-glitch]

/assign