[Review App] pods in failed state due to `Pod Predicate NodeAffinity failed`

Summary

Since pods have been assigned (via nodeSelector) to new node pools (!65938 (merged)), many pods have ended up in a failed state with the error Pod Predicate NodeAffinity failed, for example:

The problem described in this issue seems to match our problem, so it looks like we need to upgrade the nodes.

This GCP console workloads search (access required) shows all the workloads with failed pods.