Unable to expand Cloud NAT IP routing any further

In recent incident: production#3448 (closed)

We needed to add additional IP addresses to our Cloud NAT device. Luckily we were able to easily add two ip addresses that were next in line with our IP reservations. Should we need to expand any further, our terraform module may not support this.

Utilize this issue to accomplish the following:

  1. Learn how IP allocation works with respect to Cloud NAT devices - during the incident, we appeared to use IP addresses that are not technically allowed to be used in the CIDR range
  2. Determine how we can expand the set of IP's to the Cloud NAT device - currently we apply a CIDR range and a count of IP addresses - the current allocated CIDR has no free IP addresses
  3. Documentation updated to reflect how these are configured and how to expand in the future
  4. Consider expanding our current Cloud NAT IP Allocation

Marking this as high priority because the next time we run out of ports, we currently do not have an identified mitigation strategy.

Edited by John Skarbek