Fix GCP multiple user IAM policy that uses single JSON binding

Debug notes

User names have been changed to fake users for privacy.

Cloud Account User Roles

User Role Short ID Cloud Account User Cloud Account Role Created at Expires at
eabd30cd [cc3f0c15] dmurphy@gitlab.com [5504eb52] owner 2021-08-26T16:54:15+00:00 never
38c34553 [6d87b3e8] klibby@gitlab.com [5504eb52] owner 2021-08-26T16:55:17+00:00 never
e5fb3d09 [cc3f0c15] dmurphy@gitlab.com [ed1f41d9] container.admin 2021-08-26T16:54:26+00:00 never
15f23342 [6d87b3e8] klibby@gitlab.com [ed1f41d9] container.admin 2021-08-26T16:55:27+00:00 never

Only Kate Libby's container.admin role actually appears in the IAM Web UI.

cloud_account_user_role eabd30cd has an API meta data response from Google in the usual format. However I don’t see a project name in the binding JSON, so want to check if the one successful entry for Kate Libby has it.

{
  "bindings": [
    {
      "members": [
        "user:dmurphy@gitlab.com"
      ],
      "role": "roles/owner"
    }
  ],
  "etag": "BwXKeTeczZM=",
  "version": 1
}

cloud_account_user_role 15f23342 with the only actual provisioned role. No apparent differences.

{
  "bindings": [
    {
      "members": [
        "user:klibby@gitlab.com"
      ],
      "role": "roles/container.admin"
    }
  ],
  "etag": "BwXKeTvnWdg=",
  "version": 1
}

I'm modifying the database row to be able to trigger a re-provisioning with the background job.

  • Set api_meta_data column to null
  • Set flag_provisioned to 0
  • Set provisioned_at to null
  • Change state from active to provisioning-pending

After running Tinker to get list of jobs to be picked up by background job, it is not appearing in the query. After quick debug, the query looks for a value in provision_at (which is designed for delayed provisioning). After setting a manual date value in the database, the query shows results.

By trailing /srv/www/gitlabsandbox.cloud/storage/logs/cron.log, I see the role now provisioning.

[2021-08-26 21:33:02] [470ffb06-2d13-46df-bb9c-63f2149a3fd9] Cloud Account is now active
[2021-08-26 21:34:02] [c7c0ca38-1659-478d-8f05-ac04391bf91f] Provisioning Cloud Account User
[2021-08-26 21:34:02] [e7fae981-a575-4b8a-a941-28474d662823] Provisioning Cloud Account User Role
[2021-08-27 07:50:02] [da89ec18-f6d5-48b1-a2eb-26df82ffcff7] Cloud Account is now active
[2021-08-27 07:51:01] [c7975080-2f4b-4b37-96b5-75ae816be955] Provisioning Cloud Account User
[2021-08-27 07:51:02] [2278bc24-f833-44b9-be76-dc80690eeb98] Provisioning Cloud Account User Role
[2021-08-27 16:20:02] [43e5a642-4909-4da9-8e79-82e8b5aba90e] Provisioning Cloud Account User Role
[2021-08-30 16:35:02] [e5fb3d09-6859-4e85-b131-20f8e07c0a28] Provisioning Cloud Account User Role

The user role now shows up in the Web UI console. I'm repeating these steps for the remaining unprovisioned roles and will update HackyStack code soon.

After repeating the steps, it only shows the last entry for Kate Libby again. I've now connected the dots that it's a single IAM binding policy with multiple users and the current HackyStack code is overriding each previous user role.

I'm going to manually add the users in IAM for today and refactor the code afterwards.

Edited by Jeff Martin