Infrastructure setup for testnet should automate multiple instances for multiple nodes

[CMCDragonkai]

Right now Polykey-Infrastructure does not check if an existing instance is already running. It should be doing this before attempting to run more instances.

It also should have the ability to setup multiple instances depending on a fixed cluster configuration. To do this, it depends on #285 first so it can derive the NodeID during configuration.

See: https://gitlab.com/MatrixAI/Engineering/Polykey/Polykey-Infrastructure/-/merge_requests/2#note_1146016192 for more information.

Tasks

• 1. Change script's nodeId derivation from the recovery codes stored in the secret manager
• 2. Use a loop to create a separate instance for each NodeId.
• 3. Observe that the IPs are properly setup on the testnet.polykey.io
• [ ] 4. Ensure that the fixed pool of recovery codes are put into the PK source code - this will be a task on Integration Tests for testnet.polykey.com Polykey-CLI#71.

[CMCDragonkai]

@tegefaulkes please link the PR for Polykey-Infrastructure that addresses this.

[CMCDragonkai]

It's important for each node that gets created to be given their seed node list ahead of time since they can't dynamically discover each other.

This means they would need to know the IPs that we are going to be assigning the instances before we plugin and start each node.

This means you must take the set of recovery codes, map them to the instances and attached EIPs first, then construct the seed node config list, and idempotently setup the task definitions with the seed node list then starting the seed node.

[CMCDragonkai]

Alternatively we can just pass a set of seed nodes and re-use the DNS testnet.polykey.io. But you need to check if the discovery process is repeated. It should be repeated for the seed nodes.

Unlike other nodes, seed nodes are special case:

1. Proxy connections to them must always be maintained, so no node connection TTL
2. They must also exist in the NG, so they must not be expired from the NG

[tegefaulkes]

This is mostly done. We have the Polykey-infrastructure script creating multiple nodes now. I'm nut sure what 4. means.

For the seed nodes to discover each other they need to support multi-node resolving. That is currently being worked on. I don't think this can be resolved until then.

[CMCDragonkai]

Task 4 is a continuation of #269. We never got around to putting our trusted NodeIds into the source code. Create a new PR for that and merge it into staging branch.

We'll start with 2, the 2 that you already setup now, and we will update that when we finish #446 as the keypairs become Ed25519 keypairs.

This has to be done as part of MatrixAI/Polykey-CLI#71, so I'll close this.

[CMCDragonkai]

Just a note, the placement constraints was applied from service to EC2 instance ID. This was sufficient to ensure that if the setup script is re-ran, it would always start a service with respect to an instance that is always mapped from a given recovery code.