Customer creation from Sold To Contact in ContactWorker fails for invalid emails in Stg

Problem

As described in #9303 (comment 1887981729), we've seen a number of errors when attempting to create a Customer from the Sold To Contact in the Staging environment. This occurs in the ZuoraCallout::SyncResource::ContactWorker here and the ZuoraCallout::SyncResource::AccountWorker here.

The error in this case is due to an invalid email address, most commonly test@test.com being used as the Sold To Account Contact. test@test.com is not a valid email address in CDot staging due to the configuration of Devise in that environment. Anything that does not end in @gitlab.com is not valid here.

Example Details:

• Sentry Error
• BillingAccountContact in CDot

When inspecting the Zuora Accounts associated with these Contact, they appear to be related to manual testing likely originating outside of CDot. I'm seeing account names like Test CPQ Upgrade 10.28 and TC-11 UAT Bug 1.

Proposal

It might be nice to avoid these false alarms. One way we could do this is to avoid attempting to create the customer from the worker (or perhaps even in the FindOrCreateCustomerService 🤔) if the email address is invalid in the staging environment.

This is a low priority given it's happening in staging for test accounts. It doesn't cause any known problems either as the contact and/or account is synced as expected, but just doesn't end up creating a Customer for the Sold To contact.