Improve performance of SubscriptionSynchronizer to avoid timeouts

Problem

CDot experiences ActiveRecord::ConnectionFailed errors like the following occasionally. These errors occur because the transaction in the SubscriptionSynchronizer is being held longer than the idle_in_transaction_session_timeout (6s).

PQconsumeInput() FATAL:  terminating connection due to idle-in-transaction timeout (ActiveRecord::ConnectionFailed)
SSL connection has been closed unexpectedly

These may occur more often for various reasons:

  1. Zuora is responding more slowly
  2. Network latency
  3. A subscription has an especially large number of rate plans, charges, and tiers to sync

Proposal

Let's investigate ways to improve performance of the Zuora::Synchronizers::SubscriptionSynchronizer to avoid holding on the the transaction for so long, risking exceeding the timeout. There are a couple of ideas we could consider:

  1. Query for subscription data outside of the transaction
    • Move the logic for fetching subscription Rate Plans, Rate Plan Charges, and Rate Plan Charge Tiers outside of the transaction. This way the only logic that occurs within the transaction are save operations.
  2. One query for all Subscription data
    • Instead of querying for Rate Plans, Rate Plan Charges, and Rate Plan Charges in individual queries (which uses the Query action in IronBank - API reference), we could consider using the SubscriptionsByKey API or something similar which could return all this in one response. We will need to check the compatibility of this API with the query API as it could have slightly different behavior or include different attributes. For example, the all_rate_plans method has a special condition to include removed rate plans and this behavior is important for syncing so that all rate plans are synced.

Result

More efficient and timeout tolerant subscription synchronization