Skip to content

Fix CircleCI cleanup

Daniel Clemente Laboreo requested to merge clemente/fix-circleci-cleanup into master

This fixes 3 2 problems when cleaning integrations tests:

  1. nova not found:
Running integration test cleanup script with credentials from environment variables...
bin/integration-cleanup
Querying Nova for list of running integration test instances...
bin/integration-cleanup: line 21: nova: command not found
No active instances returned, nothing to clean up. Quitting.

The solution is to run the script inside the virtualenv.

  1. The OpenStack region name wasn't specified and nova didn't use GRA3. Original vs. fixed behaviour:
(venv) circleci@a384d791b3f0:~/project$ export OS_USERNAME=$OPENSTACK_USER
(venv) circleci@a384d791b3f0:~/project$ export OS_PASSWORD=$OPENSTACK_PASSWORD
(venv) circleci@a384d791b3f0:~/project$ export OS_TENANT_NAME=$OPENSTACK_TENANT
(venv) circleci@a384d791b3f0:~/project$ export OS_AUTH_URL=$OPENSTACK_AUTH_URL
(venv) circleci@a384d791b3f0:~/project$ nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+
(venv) circleci@a384d791b3f0:~/project$ export OS_REGION_NAME=$OPENSTACK_REGION
(venv) circleci@a384d791b3f0:~/project$ nova list                              
+--------------------------------------+--------------------------+--------+------------+-------------+----------------------------------------------------+
| ID                                   | Name                     | Status | Task State | Power State | Networks                                           |
+--------------------------------------+--------------------------+--------+------------+-------------+----------------------------------------------------+
| 8351ce85-d651-424d-b67c-759c58b5fa11 | edxapp-293f5ff9integ-1   | ACTIVE | -          | Running     | Ext-Net=54.36.54.75, 2001:41d0:302:1100::a:7f5f    |
| 5f0f5997-8669-45db-93b9-b6a0315d9b87 | edxapp-36b5efc1integ-1   | ACTIVE | -          | Running     | Ext-Net=54.38.244.35, 2001:41d0:302:1100::a:7fdb   |
| 4f460f7b-bad1-417f-bd5a-ba00af4465b6 | edxapp-3b47c3dcinteg-1   | ACTIVE | -          | Running     | Ext-Net=54.36.53.213, 2001:41d0:302:1100::a:7f5a   |

3. The cleanup should also run when tests fail, not only as the last step of the script. This is done with when ← this part was already merged in other PR

Note/disclaimer:

  • AWS IAM users and S3 buckets are deprovisioned as part of the tearDown process, together with other resources
  • when clicking the Cancel button, nothing will be cleaned up because CircleCI doesn't run the cleanup (instead, it finishes immediately). There's a request to add post-cancel scripts but it's not implemented. So let's avoid clicking it

Testing

  1. Re-run this PR's CircleCI build or check it if it's running or has run
  2. Check the logs, in particular the last panes in the accordion, the one called „Cleanup“ (this step comes directly from circle.yml)
  3. You should see no errors there (in previous builds we saw „nova: command not found“, etc.)
  4. Check that old instances are deleted from our OVH user (ending in …Htv). We won't see this at every run. The script says that instances older than 4 h are deleted. We can either check the log inside CircleCI (that same panel mentioned before) to see how the instances are removed, or verify the result by logging in to OVH.
  5. Check that tearDown method is called when tests fail. For that I have another PR: https://github.com/open-craft/opencraft/pull/302 . Check (if you want) that a test failure there causes the extra message added to the tearDown to be called

Merge request reports