GCP Google-provided service accounts removed with HackyStack Group Projects IAM provisioning

When using HackyStack to add an IAM user to a GCP project, there is a known issue in #69 (closed) with the IAM policy getting overriden so any other users accounts or the role for that user if they have multiple role.

It was discovered that this also overwrites/removes any project specific (direct inheritance) IAM permissions of any kind, including service accounts and Google-managed service accounts.

The affects the ability for GCP to manage services and throws error messages on clusters and likely other services.

Workaround

The service accounts still exist, however the IAM roles were removed. You will need to visit the IAM page and add the service account email addresses and map appropriate roles back to them. Note that the email addresses will not autopopulate in the dropdown list so they need to be typed in.

It was also discovered that only some service accounts are shown in the IAM web console, so be sure to check the box in the top right corner to Include Google-provided role grants otherwise it will be a "hidden" service account.

😂 45 minutes were wasted here figuring this out why the roles were not appearing.

Related Links

• User Role Previous GitLab Triage Issue (internal): it-infra#67 HackyStack GCP Group Project User Role Not Provisioned Triage
• GitLab Runbook: Add IAM User to Group Account
• Source Code: CloudAccountUserRoleService
• Source Code: CloudResourcemanagerIamUserRoleService
• GCP Troubleshooting Documentation: GKE Service Account Deleted

Runbook Excerpt

1. Run the CLI command to create a new cloud account user.

    php hacky cloud-account-user:create
   
    Which tenant should this cloud account belong to?:
    > default
   
    Which cloud account should the user be attached to?:
    > {department}-sandbox-{title}
    > Example. sales-cs-sandbox-myproject
   
    What is the email address of the user that you want to attach to the cloud account?:
    > jdoe@gitlab.com

   If you make a typo at any point, press Ctrl+C to escape and start over.

2. Run the CLI command to grant the user permissions to the account.

    php hacky cloud-account-user:create
   
    Which tenant should this cloud account belong to?:
    > default
   
    Which cloud account should the user be attached to?:
    > {department}-sandbox-{title}
    > Example. sales-cs-sandbox-myproject
   
    Which cloud account user should the role be attached to?:
    > jdoe-e5f6g7h8
   
    Which cloud account role should the user be attached to?:
    > owner