Skip to content

Small fixes to the Matrix backfilling history script pt. 2

Eric Eastwood requested to merge backfill-messages-to-matrix-follow-up2 into develop

Small fixes to the Matrix backfilling history script pt. 2

Follow-up to !2313 (merged)

Part of #2609 (closed)

Dev notes

Error spawns from mongodb-core -> lib/connection/connection.js

connection 10 to mongo-replica-01:27017 timed out, stack=MongoError: connection 10 to mongo-replica-01:27017 timed out
    at Function.MongoError.create (/opt/gitter/gitter-webapp-staging-old/node_modules/mongoose/node_modules/mongodb-core/lib/error.js:29:11)
    at Socket.<anonymous> (/opt/gitter/gitter-webapp-staging-old/node_modules/mongoose/node_modules/mongodb-core/lib/connection/connection.js:176:20)
    at Object.onceWrapper (events.js:421:28)
    at Socket.emit (events.js:315:20)
    at Socket.EventEmitter.emit (domain.js:467:12)
    at Socket._onTimeout (net.js:483:8)
    at listOnTimeout (internal/timers.js:554:17)
    at processTimers (internal/timers.js:497:7), name=MongoError

Are we getting timeouts because we're hitting max connections in Mongo?

idk

via https://dba.stackexchange.com/questions/73770/how-to-limit-the-connections-per-host-in-mongodb

> db.serverStatus().connections
{
	"current" : 314,
	"available" : 50886,
	"totalCreated" : NumberLong(468146586)
}

via https://stackoverflow.com/a/51804590/796832

> db.adminCommand({getParameter:"*"})
{
        // [...]
	"connPoolMaxConnsPerHost" : 200,
	"connPoolMaxShardedConnsPerHost" : 200,
        // The default `cursorTimeoutMillis` is `600000ms` (10 minutes), https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.cursorTimeoutMillis
	"cursorTimeoutMillis" : 600000,
        // [...]
}

Find connections by host/IP, via https://stackoverflow.com/a/55493746/796832

> db.currentOp(true).inprog.reduce(
  (accumulator, connection) => {
    ipaddress = connection.client ? connection.client.split(":")[0] : "Internal";
    accumulator[ipaddress] = (accumulator[ipaddress] || 0) + 1;
    accumulator["TOTAL_CONNECTION_COUNT"]++;
    return accumulator;
  },
  { TOTAL_CONNECTION_COUNT: 0 }
)

// Example return:
{
    "TOTAL_CONNECTION_COUNT" : 331,
    "192.168.253.72" : 8,
    "192.168.254.42" : 17,
    "127.0.0.1" : 3,
    "192.168.248.66" : 2,
    "11.178.12.244" : 2,
    "Internal" : 41,
    "3.100.12.33" : 86,
    "11.148.23.34" : 168,
    "81.127.34.11" : 1,
    "84.147.25.17" : 3
}

https://www.mongodb.com/docs/v2.4/reference/configuration-options/#maxConns

  • maxIncomingConnections
  • maxConns

No cursor timeout

We can't use .addCursorFlag('noCursorTimeout', true) because this was introduced in mongoose@4.11.0 and we're using mongoose@4.6.8 and blocked on upgrading.

Versions:

Edited by Eric Eastwood

Merge request reports