Draft: Add script to test backfilling history to matrix

Add script to test backfilling history to matrix

Part of #2609 (closed)

• Used in the demo at https://youtu.be/gSrqj2IzJew?t=523
• And the demo gif for TWIM: https://matrix.org/blog/2021/06/25/this-week-in-matrix-2021-06-25#gitter

Testing strategy

1. Delete the canonical primary alias from the room directory, DELETE http://localhost:18008/_matrix/client/r0/directory/room/%23MadLittleMods_test-history-import1%3Amy.matrix.host
2. Delete all of the bridged chat message entries: mongo gitter -> db.matricesbridgedchatmessage.remove({ matrixRoomId: db.matricesbridgedroom.findOne({ troupeId: db.troupes.findOne({ lcUri: 'madlittlemods/test-import1' })._id }).matrixRoomId })
3. Delete the bridged room entry: mongo gitter -> db.matricesbridgedroom.remove({ troupeId: db.troupes.findOne({ lcUri: 'madlittlemods/test-import1' })._id })
4. Send a message in the room again to create a new bridged Matrix room
5. DEBUG=gitter:scripts:* matrix__bridge__applicationServicePort=9001 node ./scripts/utils/backfill-existing-history-to-matrix.js --uri MadLittleMods/test-import1

Dev notes

DEBUG=gitter:scripts:* matrix__bridge__applicationServicePort=9001 node ./scripts/utils/backfill-existing-history-to-matrix.js --uri MadLittleMods/test-history-import1

DEBUG=gitter:scripts:* node ./scripts/utils/backfill-existing-history-to-matrix.js --uri EricTroupeTester/ndtrrtnnt

set DEBUG=gitter:scripts:*&&node ./scripts/utils/backfill-existing-history-to-matrix.js --uri EricTroupeTester/ndtrrtnnt

$ mongo gitter

> db.matricesbridgedroom.findOne({ troupeId: db.troupes.findOne({ lcUri: 'erictroupetester/ndtrrtnnt' })._id })
> db.matricesbridgedroom.remove({ troupeId: db.troupes.findOne({ lcUri: 'erictroupetester/ndtrrtnnt' })._id })

> db.matricesbridgeduser.findOne({ userId: db.users.findOne({ username: 'MadLittleMods' })._id })
> db.matricesbridgeduser.remove({ userId: db.users.findOne({ username: 'MadLittleMods' })._id })

DELETE http://localhost:18008/_matrix/client/r0/directory/room/%23MadLittleMods_ytjyttjy%3Amy.matrix.host

Docker

$ docker exec -it synapse /bin/bash

$ apt-get update

$ apt-get install -y less

// To get `ps aux`
$ apt-get install -y procps

$ docker logs synapse --follow

Docker compose (docker images | grep "complement-synapse" to get the image ID):

  synapse:
    container_name: 'synapse'
    image: 7b63c557e03e
    restart: always
    ports:
      - '18008:8008'
      - '18448:8448'
    volumes:
      - './scripts/docker/matrix/synapse/data/:/data/'
    entrypoint: ['/start.py']

Run the image/container standalone:

$ docker run -it --rm \
    --publish 18008:8008 \
    --publish 18448:8448 \
    --volume /Users/eric/Documents/gitlab/webapp/scripts/docker/matrix/synapse/data/:/data/ \
    --name synapse-msc2716-testing \
    7b63c557e03e

$ docker-compose --version
Docker Compose version v2.0.0-rc.1

Time forecasting to import everything

const numMessagesToImport = 141778827;
const averageBatchSendRequestTimeSeconds = 8;
const batchSize = 100;
const messagesProcessedPerSecond = batchSize / averageBatchSendRequestTimeSeconds;
const numberOfWorkers = 8;

const secondsToImportEverything = numMessagesToImport / (numberOfWorkers * messagesProcessedPerSecond);
const minutesToImportEverything = secondsToImportEverything / 60;
const hoursToImportEverything = minutesToImportEverything / 60;
const daysToImportEverything = hoursToImportEverything / 24;

console.log('hoursToImportEverything', hoursToImportEverything);
console.log('daysToImportEverything', daysToImportEverything);

Generate a lot of messages

Fetch a random word and send 50 messages for each word:

const tokens = [
  // ...
];

for (let wordIndex = 0; wordIndex < 100; wordIndex++) {
  const wordRes = await fetch(`https://cors-anywhere.herokuapp.com/https://random-words-api.vercel.app/word`, {
    method: 'GET',
    headers: {
      'Content-Type': 'application/json'
    }
  });
  const wordData = await wordRes.json();
  const word = wordData[0].word;

  const messages = [...Array(50).keys()].map((i) => (`${word} ${i}`));

  for (const [index, message] of messages.entries()) {
    const token = tokens[index % tokens.length]

    await fetch(`http://localhost:5000/api/v1/rooms/612079fa9e1321909c8bf4d2/chatMessages`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${token}`
      },
      body: JSON.stringify({
        text: message
      })
    })
  }
}

Send 50 messages:

const tokens = [
  // ...
];

const messages = [...Array(50).keys()].map((i) => (`clinic ${i}`))

for (const [index, message] of messages.entries()) {
  const token = tokens[index % tokens.length]

  await fetch(`http://localhost:5000/api/v1/rooms/612079fa9e1321909c8bf4d2/chatMessages`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${token}`
    },
    body: JSON.stringify({
      text: message
    })
  })
}