Skip to content

Make bridged Matrix room batch processing size smaller to avoid cursor timeout

Eric Eastwood requested to merge smaller-matrix-bridge-update-batch-processing-size into develop

Make bridged Matrix room batch processing size smaller to avoid cursor timeout.

If our update loop takes more than 10 minutes to process the previous batch size of 100 the cursor can idle timeout. We are just adjusting the size down so we can process the 25 batch in time and grab the next batch from the cursor. I only saw this timeout when running on production where the homeserver is real and across the ocean (US -> Europe).

{ MongoError: Cursor not found, cursor id: 402055493628
    at Function.MongoError.create (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:31:11)
    at /opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:483:72
    at authenticateStragglers (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:429:16)
    at Connection.messageHandler (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:463:5)
    at Socket.<anonymous> (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:309:22)
    at Socket.emit (events.js:198:13)
    at Socket.EventEmitter.emit (domain.js:448:20)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at Socket.Readable.push (_stream_readable.js:224:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
  name: 'MongoError',
  message: 'Cursor not found, cursor id: 402055493628',
  ok: 0,
  errmsg: 'Cursor not found, cursor id: 402055493628',
  code: 43 } 'MongoError: Cursor not found, cursor id: 402055493628\n    at Function.MongoError.create (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:31:11)\n    at /opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:483:72\n    at authenticateStragglers (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:429:16)\n    at Connection.messageHandler (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:463:5)\n    at Socket.<anonymous> (/opt/gitter/gitter-webapp/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:309:22)\n    at Socket.emit (events.js:198:13)\n    at Socket.EventEmitter.emit (domain.js:448:20)\n    at addChunk (_stream_readable.js:288:12)\n    at readableAddChunk (_stream_readable.js:269:11)\n    at Socket.Readable.push (_stream_readable.js:224:10)\n    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)'

Testing strategy

  1. Run the utility script set matrix__bridge__applicationServicePort=9001&&node ./scripts/utils/ensure-existing-bridged-matrix-rooms-up-to-date.js
Edited by Eric Eastwood

Merge request reports