Skip to content

Draft: Optimize sub-batch size for BBM

Leonardo da Rosa requested to merge 389953-optimize-batched-background-migrations-sub-batch-processing into master

What does this MR do and why?

Add initial structure for sub batch optimization

The batch optimizer now also calculates the optimal size for sub batch size.

Analysis

Case 1:

  • Number of records: 118.000
  • Delay interval: 2.minutes
  • Batch size: 1000
  • Sub batch size: 25
  • Number of batches (non-optimized): 31
  • Number of batches (optimized): 31

Sub-batch_optimization_time.svg

Chart data:
Migration Logs without sub-batch optimization 
{"severity":"INFO","time":"2023-07-05T20:21:18.648Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.39"}
{"severity":"INFO","time":"2023-07-05T20:21:23.035Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.35"}
{"severity":"INFO","time":"2023-07-05T20:21:27.434Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-05T20:21:31.869Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-05T20:21:36.256Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-05T20:21:40.700Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.42"}
{"severity":"INFO","time":"2023-07-05T20:21:45.098Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-05T20:21:49.513Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.39"}
{"severity":"INFO","time":"2023-07-05T20:21:53.954Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-05T20:21:58.601Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.62"}
{"severity":"INFO","time":"2023-07-05T20:22:03.111Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.44"}
{"severity":"INFO","time":"2023-07-05T20:22:07.522Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-05T20:22:11.998Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.42"}
{"severity":"INFO","time":"2023-07-05T20:22:16.373Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.35"}
{"severity":"INFO","time":"2023-07-05T20:22:21.713Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1200, Sub Batch Size: 25, Time Spent: 5.31"}
{"severity":"INFO","time":"2023-07-05T20:22:28.121Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1440, Sub Batch Size: 25, Time Spent: 6.38"}
{"severity":"INFO","time":"2023-07-05T20:22:35.815Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 1728, Sub Batch Size: 25, Time Spent: 7.67"}
{"severity":"INFO","time":"2023-07-05T20:22:45.085Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 2073, Sub Batch Size: 25, Time Spent: 9.23"}
{"severity":"INFO","time":"2023-07-05T20:22:56.001Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 2487, Sub Batch Size: 25, Time Spent: 10.88"}
{"severity":"INFO","time":"2023-07-05T20:23:09.116Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 2984, Sub Batch Size: 25, Time Spent: 13.09"}
{"severity":"INFO","time":"2023-07-05T20:23:24.848Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 3580, Sub Batch Size: 25, Time Spent: 15.7"}
{"severity":"INFO","time":"2023-07-05T20:23:43.677Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 4296, Sub Batch Size: 25, Time Spent: 18.79"}
{"severity":"INFO","time":"2023-07-05T20:24:06.694Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 5155, Sub Batch Size: 25, Time Spent: 22.98"}
{"severity":"INFO","time":"2023-07-05T20:24:35.650Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 6186, Sub Batch Size: 25, Time Spent: 28.92"}
{"severity":"INFO","time":"2023-07-05T20:25:10.208Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 7423, Sub Batch Size: 25, Time Spent: 34.51"}
{"severity":"INFO","time":"2023-07-05T20:25:54.414Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 8907, Sub Batch Size: 25, Time Spent: 44.16"}
{"severity":"INFO","time":"2023-07-05T20:26:48.077Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 10688, Sub Batch Size: 25, Time Spent: 53.61"}
{"severity":"INFO","time":"2023-07-05T20:27:53.365Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 12825, Sub Batch Size: 25, Time Spent: 65.22"}
{"severity":"INFO","time":"2023-07-05T20:29:10.772Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 15390, Sub Batch Size: 25, Time Spent: 77.34"}
{"severity":"INFO","time":"2023-07-05T20:30:04.794Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 18468, Sub Batch Size: 25, Time Spent: 53.96"}
{"severity":"INFO","time":"2023-07-05T20:30:09.894Z","correlation_id":"80c1d11d30ec7c49694034f065e965dd","message":"Batch Size: 22161, Sub Batch Size: 25, Time Spent: 4.97"}
Migration Logs with sub-batch optimization 
{"severity":"INFO","time":"2023-07-05T21:00:02.655Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.5"}
{"severity":"INFO","time":"2023-07-05T21:00:07.077Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-05T21:00:11.508Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-05T21:00:15.992Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.44"}
{"severity":"INFO","time":"2023-07-05T21:00:20.433Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-05T21:00:24.905Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.44"}
{"severity":"INFO","time":"2023-07-05T21:00:29.388Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.45"}
{"severity":"INFO","time":"2023-07-05T21:00:33.777Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-05T21:00:38.210Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-05T21:00:42.605Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-05T21:00:47.031Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-05T21:00:51.486Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-05T21:00:55.951Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.43"}
{"severity":"INFO","time":"2023-07-05T21:01:00.374Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.39"}
{"severity":"INFO","time":"2023-07-05T21:01:04.776Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-05T21:01:09.211Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-05T21:01:13.654Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.42"}
{"severity":"INFO","time":"2023-07-05T21:01:18.079Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-05T21:01:22.516Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-05T21:01:26.918Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-05T21:01:31.395Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1200, Sub Batch Size: 30, Time Spent: 4.44"}
{"severity":"INFO","time":"2023-07-05T21:01:35.936Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1440, Sub Batch Size: 36, Time Spent: 4.5"}
{"severity":"INFO","time":"2023-07-05T21:01:40.656Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 1728, Sub Batch Size: 43, Time Spent: 4.68"}
{"severity":"INFO","time":"2023-07-05T21:01:45.432Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 2073, Sub Batch Size: 51, Time Spent: 4.74"}
{"severity":"INFO","time":"2023-07-05T21:01:50.260Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 2487, Sub Batch Size: 61, Time Spent: 4.79"}
{"severity":"INFO","time":"2023-07-05T21:01:55.260Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 2984, Sub Batch Size: 73, Time Spent: 4.96"}
{"severity":"INFO","time":"2023-07-05T21:02:00.452Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 3580, Sub Batch Size: 87, Time Spent: 5.15"}
{"severity":"INFO","time":"2023-07-05T21:02:05.794Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 4296, Sub Batch Size: 104, Time Spent: 5.3"}
{"severity":"INFO","time":"2023-07-05T21:02:11.309Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 5155, Sub Batch Size: 124, Time Spent: 5.47"}
{"severity":"INFO","time":"2023-07-05T21:02:17.078Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 6186, Sub Batch Size: 148, Time Spent: 5.73"}
{"severity":"INFO","time":"2023-07-05T21:02:23.284Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 7423, Sub Batch Size: 177, Time Spent: 6.16"}
{"severity":"INFO","time":"2023-07-05T21:02:29.702Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 8907, Sub Batch Size: 212, Time Spent: 6.37"}
{"severity":"INFO","time":"2023-07-05T21:02:36.403Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 10688, Sub Batch Size: 254, Time Spent: 6.66"}
{"severity":"INFO","time":"2023-07-05T21:02:43.874Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 12825, Sub Batch Size: 304, Time Spent: 7.44"}
{"severity":"INFO","time":"2023-07-05T21:02:51.402Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 15390, Sub Batch Size: 364, Time Spent: 7.49"}
{"severity":"INFO","time":"2023-07-05T21:02:56.623Z","correlation_id":"8e2006afbf9c3bed02e4db9153d2da0c","message":"Batch Size: 18468, Sub Batch Size: 436, Time Spent: 5.18"}

Case 2:

  • Number of records: 765.486
  • Delay interval: 2.minutes
  • Batch size: 1000
  • Sub batch size: 25
  • Number of batches (non-optimized): 65
  • Number of batches (optimized): 47

Sub-batch-optimization-time-2_.svg

Chart data:
Migration Logs without sub-batch optimization 
{"severity":"INFO","time":"2023-07-06T17:41:02.684Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.45"}
{"severity":"INFO","time":"2023-07-06T17:41:07.093Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T17:41:11.474Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-06T17:41:15.912Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.39"}
{"severity":"INFO","time":"2023-07-06T17:41:20.437Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.5"}
{"severity":"INFO","time":"2023-07-06T17:41:24.822Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-06T17:41:29.262Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.41"}
{"severity":"INFO","time":"2023-07-06T17:41:33.705Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.42"}
{"severity":"INFO","time":"2023-07-06T17:41:38.180Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.45"}
{"severity":"INFO","time":"2023-07-06T17:41:42.586Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T17:41:46.977Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-06T17:41:51.463Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.46"}
{"severity":"INFO","time":"2023-07-06T17:41:55.865Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T17:42:00.257Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T17:42:04.758Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.46"}
{"severity":"INFO","time":"2023-07-06T17:42:09.151Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T17:42:13.532Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-06T17:42:17.987Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.43"}
{"severity":"INFO","time":"2023-07-06T17:42:22.378Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T17:42:26.838Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.44"}
{"severity":"INFO","time":"2023-07-06T17:42:32.115Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1200, Sub Batch Size: 25, Time Spent: 5.25"}
{"severity":"INFO","time":"2023-07-06T17:42:38.654Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1440, Sub Batch Size: 25, Time Spent: 6.51"}
{"severity":"INFO","time":"2023-07-06T17:42:46.342Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 1728, Sub Batch Size: 25, Time Spent: 7.66"}
{"severity":"INFO","time":"2023-07-06T17:42:55.540Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 2073, Sub Batch Size: 25, Time Spent: 9.17"}
{"severity":"INFO","time":"2023-07-06T17:43:06.485Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 2487, Sub Batch Size: 25, Time Spent: 10.92"}
{"severity":"INFO","time":"2023-07-06T17:43:19.676Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 2984, Sub Batch Size: 25, Time Spent: 13.16"}
{"severity":"INFO","time":"2023-07-06T17:43:35.513Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 3580, Sub Batch Size: 25, Time Spent: 15.81"}
{"severity":"INFO","time":"2023-07-06T17:43:54.884Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 4296, Sub Batch Size: 25, Time Spent: 19.34"}
{"severity":"INFO","time":"2023-07-06T17:44:19.092Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 5155, Sub Batch Size: 25, Time Spent: 24.17"}
{"severity":"INFO","time":"2023-07-06T17:44:48.255Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 6186, Sub Batch Size: 25, Time Spent: 29.12"}
{"severity":"INFO","time":"2023-07-06T17:45:22.927Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 7423, Sub Batch Size: 25, Time Spent: 34.63"}
{"severity":"INFO","time":"2023-07-06T17:46:05.997Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 8907, Sub Batch Size: 25, Time Spent: 43.01"}
{"severity":"INFO","time":"2023-07-06T17:46:59.122Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 10688, Sub Batch Size: 25, Time Spent: 53.07"}
{"severity":"INFO","time":"2023-07-06T17:48:05.095Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 12825, Sub Batch Size: 25, Time Spent: 65.92"}
{"severity":"INFO","time":"2023-07-06T17:49:24.612Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 15390, Sub Batch Size: 25, Time Spent: 79.45"}
{"severity":"INFO","time":"2023-07-06T17:50:59.778Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 18468, Sub Batch Size: 25, Time Spent: 95.12"}
{"severity":"INFO","time":"2023-07-06T17:52:54.199Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22161, Sub Batch Size: 25, Time Spent: 114.37"}
{"severity":"INFO","time":"2023-07-06T17:55:10.165Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 26593, Sub Batch Size: 25, Time Spent: 135.92"}
{"severity":"INFO","time":"2023-07-06T17:57:26.666Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 26593, Sub Batch Size: 25, Time Spent: 136.44"}
{"severity":"INFO","time":"2023-07-06T17:59:35.999Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 25263, Sub Batch Size: 25, Time Spent: 129.27"}
{"severity":"INFO","time":"2023-07-06T18:01:35.217Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 23300, Sub Batch Size: 25, Time Spent: 119.17"}
{"severity":"INFO","time":"2023-07-06T18:03:25.195Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21700, Sub Batch Size: 25, Time Spent: 109.91"}
{"severity":"INFO","time":"2023-07-06T18:05:12.646Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21035, Sub Batch Size: 25, Time Spent: 107.4"}
{"severity":"INFO","time":"2023-07-06T18:06:58.494Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21035, Sub Batch Size: 25, Time Spent: 105.79"}
{"severity":"INFO","time":"2023-07-06T18:08:46.722Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21035, Sub Batch Size: 25, Time Spent: 108.17"}
{"severity":"INFO","time":"2023-07-06T18:10:34.232Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21035, Sub Batch Size: 25, Time Spent: 107.44"}
{"severity":"INFO","time":"2023-07-06T18:12:19.709Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 21035, Sub Batch Size: 25, Time Spent: 105.41"}
{"severity":"INFO","time":"2023-07-06T18:14:14.343Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.57"}
{"severity":"INFO","time":"2023-07-06T18:16:09.546Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 115.15"}
{"severity":"INFO","time":"2023-07-06T18:17:58.624Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 109.02"}
{"severity":"INFO","time":"2023-07-06T18:19:43.806Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 105.13"}
{"severity":"INFO","time":"2023-07-06T18:21:36.457Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 112.59"}
{"severity":"INFO","time":"2023-07-06T18:23:30.770Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.26"}
{"severity":"INFO","time":"2023-07-06T18:25:24.700Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 113.87"}
{"severity":"INFO","time":"2023-07-06T18:27:19.396Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.64"}
{"severity":"INFO","time":"2023-07-06T18:29:13.202Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 113.74"}
{"severity":"INFO","time":"2023-07-06T18:31:08.235Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.97"}
{"severity":"INFO","time":"2023-07-06T18:33:03.040Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.74"}
{"severity":"INFO","time":"2023-07-06T18:34:57.525Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.43"}
{"severity":"INFO","time":"2023-07-06T18:36:52.083Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 114.49"}
{"severity":"INFO","time":"2023-07-06T18:38:42.584Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 110.44"}
{"severity":"INFO","time":"2023-07-06T18:40:27.344Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 104.71"}
{"severity":"INFO","time":"2023-07-06T18:42:11.889Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 104.49"}
{"severity":"INFO","time":"2023-07-06T18:43:56.516Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 22453, Sub Batch Size: 25, Time Spent: 104.58"}
{"severity":"INFO","time":"2023-07-06T18:44:34.582Z","correlation_id":"aa1dd9387fdf0d2d5da14a17c3cc80de","message":"Batch Size: 23966, Sub Batch Size: 25, Time Spent: 38.01"}
Migration Logs with sub-batch optimization 
{"severity":"INFO","time":"2023-07-06T19:11:50.812Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T19:11:55.218Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T19:11:59.616Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:04.005Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:08.401Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:12.794Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:17.164Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.35"}
{"severity":"INFO","time":"2023-07-06T19:12:21.743Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.55"}
{"severity":"INFO","time":"2023-07-06T19:12:26.443Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.67"}
{"severity":"INFO","time":"2023-07-06T19:12:30.824Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.35"}
{"severity":"INFO","time":"2023-07-06T19:12:35.223Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:39.600Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.35"}
{"severity":"INFO","time":"2023-07-06T19:12:44.080Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.45"}
{"severity":"INFO","time":"2023-07-06T19:12:48.511Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.4"}
{"severity":"INFO","time":"2023-07-06T19:12:52.935Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:12:57.319Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.36"}
{"severity":"INFO","time":"2023-07-06T19:13:01.724Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T19:13:06.131Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.38"}
{"severity":"INFO","time":"2023-07-06T19:13:10.526Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.37"}
{"severity":"INFO","time":"2023-07-06T19:13:15.028Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1000, Sub Batch Size: 25, Time Spent: 4.48"}
{"severity":"INFO","time":"2023-07-06T19:13:19.525Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1200, Sub Batch Size: 30, Time Spent: 4.42"}
{"severity":"INFO","time":"2023-07-06T19:13:24.028Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1440, Sub Batch Size: 36, Time Spent: 4.47"}
{"severity":"INFO","time":"2023-07-06T19:13:28.729Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 1728, Sub Batch Size: 43, Time Spent: 4.66"}
{"severity":"INFO","time":"2023-07-06T19:13:33.447Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 2073, Sub Batch Size: 51, Time Spent: 4.68"}
{"severity":"INFO","time":"2023-07-06T19:13:38.297Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 2487, Sub Batch Size: 61, Time Spent: 4.8"}
{"severity":"INFO","time":"2023-07-06T19:13:43.236Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 2984, Sub Batch Size: 73, Time Spent: 4.9"}
{"severity":"INFO","time":"2023-07-06T19:13:48.339Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 3580, Sub Batch Size: 87, Time Spent: 5.07"}
{"severity":"INFO","time":"2023-07-06T19:13:53.616Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 4296, Sub Batch Size: 104, Time Spent: 5.23"}
{"severity":"INFO","time":"2023-07-06T19:13:59.045Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 5155, Sub Batch Size: 124, Time Spent: 5.4"}
{"severity":"INFO","time":"2023-07-06T19:14:04.693Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 6186, Sub Batch Size: 148, Time Spent: 5.61"}
{"severity":"INFO","time":"2023-07-06T19:14:10.638Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 7423, Sub Batch Size: 177, Time Spent: 5.9"}
{"severity":"INFO","time":"2023-07-06T19:14:16.930Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 8907, Sub Batch Size: 212, Time Spent: 6.26"}
{"severity":"INFO","time":"2023-07-06T19:14:23.551Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 10688, Sub Batch Size: 254, Time Spent: 6.59"}
{"severity":"INFO","time":"2023-07-06T19:14:30.534Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 12825, Sub Batch Size: 304, Time Spent: 6.95"}
{"severity":"INFO","time":"2023-07-06T19:14:38.122Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 15390, Sub Batch Size: 364, Time Spent: 7.55"}
{"severity":"INFO","time":"2023-07-06T19:14:46.145Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 18468, Sub Batch Size: 436, Time Spent: 7.99"}
{"severity":"INFO","time":"2023-07-06T19:14:55.189Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 22161, Sub Batch Size: 523, Time Spent: 9.01"}
{"severity":"INFO","time":"2023-07-06T19:15:05.519Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 26593, Sub Batch Size: 627, Time Spent: 10.29"}
{"severity":"INFO","time":"2023-07-06T19:15:16.839Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 31911, Sub Batch Size: 752, Time Spent: 11.28"}
{"severity":"INFO","time":"2023-07-06T19:15:29.442Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 38293, Sub Batch Size: 902, Time Spent: 12.56"}
{"severity":"INFO","time":"2023-07-06T19:15:43.414Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 45951, Sub Batch Size: 1082, Time Spent: 13.93"}
{"severity":"INFO","time":"2023-07-06T19:15:59.101Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 55141, Sub Batch Size: 1298, Time Spent: 15.65"}
{"severity":"INFO","time":"2023-07-06T19:16:17.347Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 66169, Sub Batch Size: 1557, Time Spent: 18.2"}
{"severity":"INFO","time":"2023-07-06T19:16:38.244Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 79402, Sub Batch Size: 1868, Time Spent: 20.85"}
{"severity":"INFO","time":"2023-07-06T19:17:02.199Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 95282, Sub Batch Size: 2241, Time Spent: 23.91"}
{"severity":"INFO","time":"2023-07-06T19:17:29.165Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 114338, Sub Batch Size: 2689, Time Spent: 26.92"}
{"severity":"INFO","time":"2023-07-06T19:17:44.289Z","correlation_id":"75bf39ee3ed8ce38f2d91480edbd54df","message":"Batch Size: 137205, Sub Batch Size: 3226, Time Spent: 15.08"}

Case 3:

  • Number of records: 765.486
  • Delay interval: 2.minutes
  • Batch size: 10.000
  • Sub batch size: 750
  • Number of batches (non-optimized): 33
  • Number of batches (optimized): 33

Sub-batch-optimization-time-3_.svg

Chart data:
Migration Logs without sub-batch optimization 
{"severity":"INFO","time":"2023-07-06T19:52:59.451Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:53:02.925Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:53:06.426Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.48"}
{"severity":"INFO","time":"2023-07-06T19:53:09.964Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.51"}
{"severity":"INFO","time":"2023-07-06T19:53:13.726Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.73"}
{"severity":"INFO","time":"2023-07-06T19:53:17.214Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.46"}
{"severity":"INFO","time":"2023-07-06T19:53:20.686Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.45"}
{"severity":"INFO","time":"2023-07-06T19:53:24.183Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.47"}
{"severity":"INFO","time":"2023-07-06T19:53:27.709Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.5"}
{"severity":"INFO","time":"2023-07-06T19:53:31.206Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.47"}
{"severity":"INFO","time":"2023-07-06T19:53:34.664Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T19:53:38.097Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.41"}
{"severity":"INFO","time":"2023-07-06T19:53:41.541Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.42"}
{"severity":"INFO","time":"2023-07-06T19:53:45.004Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:53:48.454Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T19:53:51.924Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:53:55.383Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:53:58.851Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:54:02.323Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T19:54:05.773Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T19:54:09.840Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 12000, Sub Batch Size: 750, Time Spent: 4.04"}
{"severity":"INFO","time":"2023-07-06T19:54:14.812Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 14400, Sub Batch Size: 750, Time Spent: 4.94"}
{"severity":"INFO","time":"2023-07-06T19:54:20.856Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 17280, Sub Batch Size: 750, Time Spent: 6.01"}
{"severity":"INFO","time":"2023-07-06T19:54:28.071Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 20736, Sub Batch Size: 750, Time Spent: 7.18"}
{"severity":"INFO","time":"2023-07-06T19:54:36.684Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 24883, Sub Batch Size: 750, Time Spent: 8.58"}
{"severity":"INFO","time":"2023-07-06T19:54:47.003Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 29859, Sub Batch Size: 750, Time Spent: 10.29"}
{"severity":"INFO","time":"2023-07-06T19:54:59.484Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 35830, Sub Batch Size: 750, Time Spent: 12.45"}
{"severity":"INFO","time":"2023-07-06T19:55:14.437Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 42996, Sub Batch Size: 750, Time Spent: 14.92"}
{"severity":"INFO","time":"2023-07-06T19:55:33.093Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 51595, Sub Batch Size: 750, Time Spent: 18.62"}
{"severity":"INFO","time":"2023-07-06T19:55:54.287Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 61914, Sub Batch Size: 750, Time Spent: 21.16"}
{"severity":"INFO","time":"2023-07-06T19:56:19.412Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 74296, Sub Batch Size: 750, Time Spent: 25.09"}
{"severity":"INFO","time":"2023-07-06T19:56:49.418Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 89155, Sub Batch Size: 750, Time Spent: 29.97"}
{"severity":"INFO","time":"2023-07-06T19:57:19.774Z","correlation_id":"65aaec913235be9ac73c36764ae98e82","message":"Batch Size: 106986, Sub Batch Size: 750, Time Spent: 30.32"}
Migration Logs with sub-batch optimization 
{"severity":"INFO","time":"2023-07-06T20:10:05.777Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T20:10:09.273Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.46"}
{"severity":"INFO","time":"2023-07-06T20:10:12.733Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T20:10:16.204Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T20:10:19.671Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T20:10:23.089Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.39"}
{"severity":"INFO","time":"2023-07-06T20:10:26.545Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T20:10:29.988Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.42"}
{"severity":"INFO","time":"2023-07-06T20:10:33.425Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.41"}
{"severity":"INFO","time":"2023-07-06T20:10:36.860Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.41"}
{"severity":"INFO","time":"2023-07-06T20:10:40.289Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.41"}
{"severity":"INFO","time":"2023-07-06T20:10:43.788Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.48"}
{"severity":"INFO","time":"2023-07-06T20:10:47.253Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T20:10:50.725Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.45"}
{"severity":"INFO","time":"2023-07-06T20:10:54.179Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.43"}
{"severity":"INFO","time":"2023-07-06T20:10:57.642Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.44"}
{"severity":"INFO","time":"2023-07-06T20:11:01.071Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.4"}
{"severity":"INFO","time":"2023-07-06T20:11:04.501Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.41"}
{"severity":"INFO","time":"2023-07-06T20:11:07.977Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.45"}
{"severity":"INFO","time":"2023-07-06T20:11:11.500Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 10000, Sub Batch Size: 750, Time Spent: 3.5"}
{"severity":"INFO","time":"2023-07-06T20:11:15.355Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 12000, Sub Batch Size: 900, Time Spent: 3.82"}
{"severity":"INFO","time":"2023-07-06T20:11:19.732Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 14400, Sub Batch Size: 1080, Time Spent: 4.34"}
{"severity":"INFO","time":"2023-07-06T20:11:24.624Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 17280, Sub Batch Size: 1296, Time Spent: 4.86"}
{"severity":"INFO","time":"2023-07-06T20:11:30.229Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 20736, Sub Batch Size: 1555, Time Spent: 5.57"}
{"severity":"INFO","time":"2023-07-06T20:11:36.721Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 24883, Sub Batch Size: 1866, Time Spent: 6.46"}
{"severity":"INFO","time":"2023-07-06T20:11:44.248Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 29859, Sub Batch Size: 2239, Time Spent: 7.49"}
{"severity":"INFO","time":"2023-07-06T20:11:53.225Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 35830, Sub Batch Size: 2686, Time Spent: 8.94"}
{"severity":"INFO","time":"2023-07-06T20:12:03.817Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 42996, Sub Batch Size: 3223, Time Spent: 10.55"}
{"severity":"INFO","time":"2023-07-06T20:12:15.986Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 51595, Sub Batch Size: 3867, Time Spent: 12.13"}
{"severity":"INFO","time":"2023-07-06T20:12:30.172Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 61914, Sub Batch Size: 4480, Time Spent: 14.12"}
{"severity":"INFO","time":"2023-07-06T20:12:47.054Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 74296, Sub Batch Size: 4480, Time Spent: 16.81"}
{"severity":"INFO","time":"2023-07-06T20:13:06.758Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 89155, Sub Batch Size: 4480, Time Spent: 19.67"}
{"severity":"INFO","time":"2023-07-06T20:13:27.282Z","correlation_id":"a2e5fb3a77d0c2dd039f74f6edf617fa","message":"Batch Size: 106986, Sub Batch Size: 4836, Time Spent: 20.48"}

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #389953 (closed)

Edited by Leonardo da Rosa

Merge request reports