Skip to content

Try again and resume if the import fails (like a cursor timeout) - Small fixes to the Matrix backfilling history script pt. 6

Eric Eastwood requested to merge backfill-messages-to-matrix-follow-up6 into develop

Small fixes to the Matrix backfilling history script pt. 6: Try again and resume if the import fails (like a cursor timeout)

Follow-up to !2313 (merged)

Part of #2609 (closed)

Error occurred while backfilling events: MongoError: Cursor not found, cursor id: 240894090023
    at Function.MongoError.create (/opt/gitter/gitter-webapp-staging/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:31:11)
    at /opt/gitter/gitter-webapp-staging/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:483:72
    at authenticateStragglers (/opt/gitter/gitter-webapp-staging/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:429:16)
    at Connection.messageHandler (/opt/gitter/gitter-webapp-staging/node_modules/mongoose/node_modules/mongodb-core/lib/connection/pool.js:463:5)
    at Socket.<anonymous> (/opt/gitter/gitter-webapp-staging/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:309:22)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:467:12)
    at addChunk (internal/streams/readable.js:309:12)
    at readableAddChunk (internal/streams/readable.js:284:9)
    at Socket.Readable.push (internal/streams/readable.js:223:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)

Todo

  • Add some sort of abort mechanism for the currently running lanes before we restart any import process. We need to be careful not to have two processes importing the same room.
  • Add a stat to the dashboard to indicate how many times the import process has been restarted
Edited by Eric Eastwood

Merge request reports