Integration Tests for `testnet.polykey.com`

[emmacasolin]

Specification

We need a suite of tests to cover interacting with a deployed agent, which we can do using testnet.polykey.io. These tests need to cover various different connection scenarios:

• Connecting to a deployed agent as a seed node during startup
• Pinging a deployed agent
• Using the deployed agent as a signaller (and eventually a relay once this is implemented)
• Tests for when the deployed agent is contacting an agent behind a NAT (NAT-Traversal Testing with testnet.polykey.io Polykey#159)
• Any bugs that are discovered during the above tasks

These tests should go into their own subdirectory tests/testnet and should not be run with the other tests. They should be disabled in our jest config and should only run when explicitly called (which will happen during the integration stage of our pipelines).

Required tests:

• tests/testnet/testnetConnection.test.ts
  • Can connect to the testnet
    • Within a reasonable amount of time
    • Without errors/shutting down the local agent
    • Without errors/shutting down the testnet
  • Can disconnect from the testnet
    • Within a reasonable amount of time
    • Without errors/shutting down the local agent
    • Without errors/shutting down the testnet
  • Can reconnect to the testnet
    • Able to handle different node ids (testnet is a cluster of nodes)
• tests/testnet/testnetPing.test.ts
  • Can ping the testnet
    • Able to handle different node ids (testnet is a cluster of nodes)
  • Can ping another node via the testnet (signaling)
  • Can ping another node via the testnet (relay)
  • Can attempt to ping another node that doesn't exist
    • Without shutting down the testnet
• tests/testnet/testnetNAT.test.ts
  • Can ping a node that is behind endpoint-independent NAT via the testnet
    • From a node that is not behind a NAT (DMZ)
    • From a node that is behind endpoint-independent NAT
    • From a node that is behind endpoint-dependent NAT
  • Can ping a node that is behind endpoint-dependent NAT via the testnet
    • From a node that is not behind a NAT (DMZ)
    • From a node that is behind endpoint-independent NAT
    • From a node that is behind endpoint-dependent NAT
• Should also incorporate tests from Testnet Deployment Polykey#326 (comment)

Additional context

• NAT-Traversal Testing with testnet.polykey.io Polykey#159 - NAT traversal tests against testnet.polykey.io
• Connection dropped/timed out when connecting to deployed agent Polykey#414 - Issue for failed connections to/from a deployed agent
• Continuous connection attempts between a deployed seed node and local agent Polykey#415 - Issue for continuous re-connection bug when connecting to deployed seed node
• Starting Connection Forward infinite loop Polykey#413 - Issue for potential infinite loop when connecting to deployed seed node

Tasks

• 1. Attempt connections to the deployed seed node and create issues for all bugs discovered (and resolve them)
• 2. Create tests for simple connections to testnet.polykey.io
  • 1 node connected to testnet.polykey.io and maintains connection
  • 2 nodes connected to testnet.polykey.io and can ping each other (they will have the same IP but different ports)
• [ ] 3. Create tests for edge cases and previous bugs - most edge cases will go to the simulation suite
• [ ] 4. Create tests for connecting to a deployed seed node from behind a NAT - this can only be done as part of a simulation suite, since we don't control host firewalls
• 5. Finish off all diagrams as part of NAT testing Add diagrams to NAT Traversal tests Polykey#388
• 6. Add new INFO level logs for situations where connections are going into stopping. This is next to the debug logs.
• [ ] 7. Add a logging debug filter to command line arguments to take a regular expression. Should be global option like --log='/regex/'. - not relevant to integration testing, the js-logger does support REGEX filtering, but the PK CLI currently doesn't have this option.
• [ ] 8. Use https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerOverride.html to be able to easily try different debugging levels and filters. - cannot use container overrides in services, can be done as part of tasks though, will have to just redeploy service with different task definition each time
• [ ] 9. agent status command needs to display useful information like the polykey version and other useful statistics like active connections, number of node graph entries etc etc.
• [ ] 10. --client-host needs to support host names. - this is pending a change to being able to use PolykeyClient to connect to a host name - which would require using the DNS-SD SRV records. This still needs to be specced out how this would work because in some cases you want to connect to a SINGLE Node, in other cases you are "discovering" a node to connect to, but it's not relevant to this epic.
• 11. EC2 setup with idempotency
• 12. Multi Node Setup on AWS
• 13. Recovery Code Pool on AWS
• 14. Multi-Host DNS resolution
• 15. Multi Node Resolver
• 16. NodeGraph KeyPath to lift Host and Port to the key path
• 17. Put the trusted testnet seed nodes into the src/config.ts - from Infrastructure setup for testnet should automate multiple instances for multiple nodes Polykey#488

Emergent bugs

• 1. Seed node is failing to establish a reverse connection back to the connecting node. The problem is somewhere in ConnectionReverse.start() between Starting Connection Reverse and Started Connection Reverse. Start by testing locally.
• 2. Fixing the Random Process Termination
  • Integration Tests for testnet.polykey.com #71
  • Integration Tests for testnet.polykey.com #71
• 3. Verify that new timeouts are being respected
  • Integration Tests for testnet.polykey.com #71

[CMCDragonkai]

Seems like our tests aren't simulating long-running behaviour. This is unfortunate, because certain bugs only became apparent after running the nodes on the testnet for longer than 1 hr.

We might need to increase our modelling tooling to help with correctness. As well as monitoring tools/remote debugging tools applied to our testnet to enable observation of memory and CPU cycles. Nodejs provides this via their debugging port. This could be enabled on our testnet nodes. Possibly bootstrapping off the client port, and related to MatrixAI/Polykey#412.

[emmacasolin]

I'm also noticing that the re-connection attempts discussed in MatrixAI/Polykey#413 and MatrixAI/Polykey#415 don't appear when I use the same setup locally (two local agents where the one that gets started first is set as a seed node for the second one), so even if we had tests to simulate long-running behaviour they may not have picked up these issues since they may be limited to the testnet environment.

[CMCDragonkai]

Also unlike our NAT tests which use conditional testing with describeIf and testIf, these tests are conditional on a certain stage in our pipeline. We may continue to use this technique rather than explicitly excluding the tests in the jest config (or using explicit group tagging https://stackoverflow.com/questions/50171932/run-jest-test-suites-in-groups and https://morioh.com/p/33c2bd031589).

This approach means you use describeIf as well, but then depend on a condition that is only available via environment variables. This would be similar to @tegefaulkes work with docker integration tests which rely on a special test command environment variable.

We could re-use NODE_ENV or create our own env variable that is then funnelled into the jest.config.js as global parameters.

[CMCDragonkai]

The only difference in the testnet environment is running in a docker container. If the docker container is producing different behaviour, that has to be illuminated with the docker integration testing MatrixAI/Polykey#407.

---

Caveat: when using testnet.polykey.io you are using NLBs as well... so that adds extra complexity. But this is why we are sticking to the public IPs first.

[CMCDragonkai]

@emmacasolin you can always run your own docker container locally and set that up as your local "seed node" and have it continuously run while you write tests against it.

Just make sure to feed it all the required env variables, or the parameters and mount the necessary namespaces. Meet with @tegefaulkes about this, he's already doing this currently.

[CMCDragonkai]

Also unlike our NAT tests which use conditional testing with describeIf and testIf, these tests are conditional on a certain stage in our pipeline. We may continue to use this technique rather than explicitly excluding the tests in the jest config (or using explicit group tagging https://stackoverflow.com/questions/50171932/run-jest-test-suites-in-groups and https://morioh.com/p/33c2bd031589).

This approach means you use describeIf as well, but then depend on a condition that is only available via environment variables. This would be similar to @tegefaulkes work with docker integration tests which rely on a special test command environment variable.

We could re-use NODE_ENV or create our own env variable that is then funnelled into the jest.config.js as global parameters.

If we continue down the path of reusing describeIf, it would be optimal to move our imports to be asynchronous under the describe. That would be similar to our bin commands where we use dynamic imports. Problem is, describe doesn't support asynchronous callbacks. A long term solution is to use top-level await when it becomes available in jest: jestjs/jest#2235 (comment)

[emmacasolin]

I'm also noticing that the re-connection attempts discussed in MatrixAI/Polykey#413 and MatrixAI/Polykey#415 don't appear when I use the same setup locally (two local agents where the one that gets started first is set as a seed node for the second one), so even if we had tests to simulate long-running behaviour they may not have picked up these issues since they may be limited to the testnet environment.

This is actually incorrect. I left my local setup going in the background for several hours and both agents had attempted multiple node connections to the other agent when I checked back (so many that they were cut off). At one point when I checked one of the agents was displaying these logs (rapidly) and the other was silent but now both of them are silent. This might be from the refresh buckets queue if this is something that happens every hour? I'm going to try reducing the time between refreshes and adding timestamps to the logs.

As a side note, I think these logs appear to be infinite and constant on the testnet because it's a lot slower than my local machine, so it's only able to attempt a connection every 20 seconds, and since it takes so long to get through them it's refreshing them again by the time it's finished.

[emmacasolin]

I think I know what the main issue causing our "infinite loop" is. The timeout for opening a forward proxy connection is 20 seconds, so if we try to connect to a node that is offline it blocks the refresh buckets queue for 20 seconds. The refresh timer for the refresh buckets queue is an hour (i.e. buckets are added to the queue every hour). 1 hour / 20 seconds is 180 nodes per hour if we try to connect to an offline node for every bucket (which is the case if we only have one node in our node graph and it's offline). Since there are 256 buckets, this means we won't get through all of the buckets within the hour, and buckets will begin to be added to the queue again at the same rate that they're removed. So the queue will have 256-180=76 buckets in it forever (until the node in our node graph comes back online).

I'm not sure if this blocks the entire event loop as well, in which case this is definitely a problem.

[tegefaulkes]

The refresh bucket and ping node queues work asynchronously in the background. I don't think it will block the event loop.

The excessive contacting of nodes as part of the refresh bucket queue is not ideal. I think we do need to optimise this but the problem is, how? We need a way to determine if we are not gaining any new information and just stop refreshing buckets for a while. But right now the real problem is that we're attempting to contact the same offline node over and over again. Right now two things come to mind to address this.

1. Just use a smaller timeout. 20 seconds seems a bit much.
2. Track the last time we tried to contact a node and maybe the attempts. If we keep trying to contact it then we can skip it for a period of time.

Just a note that more aggressive removal of nodes from the node graph such as removing a node if we see it's offline would fix this. However this will lead to use removing nodes that may be temporarily offline. Or worse, if a node's network goes down it will clean out it's nodeGraph.

[CMCDragonkai]

Pleas create a new PR to tackle this issue, you may wish to incorporate the subissues within this epic too.