Corrective Action: Added column to `members` table causes 500 errors to be returned in some cases

An incident, 2022-01-18: autcomplete_sources returning 500 (gitlab-com/gl-infra/production#6182 - closed), was raised due to 500 being returned from autcomplete_sources, as well as other methods, generating

PG::SyntaxError: ERROR:  each UNION query must have the same number of columns
LINE 5: (SELECT "members".* FROM "members" LEFT OUTER JOIN "users" O...

This was determined to be because of a column, member_namespace_id, which was added to the members table.

As the change was rolled out, the cached_column_list list used by the UNION queries were failing on nodes that had not been updated yet. Re-cylcing the pods fixed the issue.

We need to either make the member group queries more resilient when a column is added (see gitlab-com/gl-infra/production#6182 (comment 814252765)) or

figure out why verify_select_values! didn't detect this in https://gitlab.com/gitlaborg/gitlab/blob/2a7cd9cd8885d86121cfb74f650a9e55460e5619/lib/gitlab/sql/set_operator.rb#L59

gitlab-com/gl-infra/production#6182 (comment 814252765)

But based on this, gitlab-com/gl-infra/production#6182 (comment 834634660), it seems non-trivial

Edited by Brett Walker