changes for controller HA

[raman-mann]

What does this PR do?

Support controller HA in openstack

Do you have any concerns with this PR?

NA

How can the reviewer verify this PR?

By deploying openstack with multiple controller nodes.

Any background context you want to provide?

NA

Screenshots or logs (if appropriate)

NA

Questions:

• Have you connected this PR to the issue it resolves?
  Controller HA #12

• Does the documentation need an update?
  Yes

• Does this add new Python dependencies?
  No

• Have you added unit or functional tests for this PR?
  No

• Does this patch update any configuration files?
  No

[skulasekar]

Build finished.

[bo-quan]

@raman-mann Could you append it with CI related changes so that we can validate this patch via CI?

[RandyLevensalor]

Additional pep8 issues were added in this commit.

[RandyLevensalor]

Run Tests

[RandyLevensalor]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

Aricent thinks the CI issue is not caused by the PR.

[mansi-jain2]

CI script has been modified to add a new controller node for HA testing, and in the logs we can see SSH failure to this 4th node.

TASK [Deploy - iaas_launch.py -d *** This will run for around an hour without output] ***
21:27:23 fatal: [10.197.113.106]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh:

@spisarski : Can you please check, if CI script changes need any further modification

[bo-quan]

Run Tests

[bo-quan]

Bo will work with Steve and Randy to see if CI script needs further change to resolve this CI failure.

[bo-quan]

Run Tests

[bo-quan]

Run Tests

[RandyLevensalor]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

The 4th vm appeared sometimes to be deploying to vdb instead of vda, causing its creation to fail. Need to dig further to understand the reason.

[skulasekar]

Run Tests

[skulasekar]

Run Tests

[bo-quan]

@raman-mann @sbhojpur After more investigation, I found that the 4th vm creation failure was due to a project quota issue. Could you please increase the project quota in ci/snaps/snaps_os_tmplt.yaml file:

     quotas:

•      ram: 102400
  

•      cores: 40
  

•      ram: 128000
  

•      cores: 48
  

[bo-quan]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

Run Tests

[bo-quan]

With the quota patch, CI passed the vm creation issue, however, it ran into the following error:

01:36:15 "EXECUTING SET HOSTS PLAY",
01:36:15 "kolla_sethosts.yaml",
01:36:15 "{'BASE_FILE_PATH': '/tmp/snaps-openstack/snaps_openstack/packages/source/', 'PROXY_DATA_FILE': '/tmp/snaps-openstack/snaps_openstack/ansible_p/ansible_utils/proxy_data.yaml', 'VARIABLE_FILE': '/tmp/snaps-openstack/snaps_openstack/ansible_p/ansible_utils/variable.yaml', 'target': '10.1.0.13', 'host_name': 'COMPUTE22557'}",
01:36:15 "FAILED IN SETTING HOSTS FILE"

@spisarski @RandyLevensalor does the error above ring a bell?

[spisarski]

@bo-quan - Sorry, it has been awhile since I've been in this code thus have never encountered the problem. Please let me know if you need assistance debugging.

[bo-quan]

@raman-mann @sbhojpur If you have a chance (since I'll be consumed in another higher priority task within the next few days), could you take a look at the CI failure:

01:36:15 "EXECUTING SET HOSTS PLAY",
01:36:15 "kolla_sethosts.yaml",
01:36:15 "{'BASE_FILE_PATH': '/tmp/snaps-openstack/snaps_openstack/packages/source/', 'PROXY_DATA_FILE': '/tmp/snaps-openstack/snaps_openstack/ansible_p/ansible_utils/proxy_data.yaml', 'VARIABLE_FILE': '/tmp/snaps-openstack/snaps_openstack/ansible_p/ansible_utils/variable.yaml', 'target': '10.1.0.13', 'host_name': 'COMPUTE22557'}",
01:36:15 "FAILED IN SETTING HOSTS FILE"

in https://snapsci.cablelabs.com:8443/job/snaps-openstack/127/console