WIP: Inspect Mongo Oplog

Inspect Mongo oplog.

Good docs,

• https://dzone.com/articles/introduction-replication
• https://blog.pythian.com/exploring-operations-log-mongodb/
• https://docs.mongodb.com/manual/core/replica-set-oplog/

$ mongo mongo-replica-01.prod.gitter
TroupeReplicaSet:PRIMARY> db.getReplicationInfo()
{
        "logSizeMB" : 1024,
        "usedMB" : 1032.28,
        "timeDiff" : 22649,
        "timeDiffHours" : 6.29,
        "tFirst" : "Wed Apr 04 2018 08:51:19 GMT-0500 (Central Standard Time)",
        "tLast" : "Wed Apr 04 2018 15:08:48 GMT-0500 (Central Standard Time)",
        "now" : "Wed Apr 04 2018 15:08:47 GMT-0500 (Central Standard Time)"
}

---

Spawned out of investigation of oplog window size becoming too small, https://app.datadoghq.com/monitors#555843?group=all&from_ts=1522783781920&to_ts=1522870181920

The oplog window size is the amount of database operations you can buffer before hitting a hard limit. The reason that the window size is still small even after the inserts/deletions have occurred is because the data is still propagating into the actual collections (can only process so many operations at a time). Thanks @northrup for helping me understand this!

@northrup created a snapshot of the box while the oplog was recovering if we want to investigate further, https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Snapshots:visibility=private;search=snap-0fbf778ccdd2f834c;sort=desc:snapshotId

---

• https://app.datadoghq.com/dash/191654/everything-mongodb?live=true&page=0&is_auto=false&from_ts=1522788833379&to_ts=1522875233379&tile_size=m&fullscreen=false
• https://app.datadoghq.com/dash/760690/2018-4-4-situation-room-mongo-oplog-window-size-too-small?live=true&page=0&is_auto=false&from_ts=1522790002942&to_ts=1522876402942&tile_size=m&fullscreen=false

The query count seems like pretty normal load to normal times 🤔. So it probably means that there was one giant query that spiked the oplog which splits it into little idempotent pieces.

Looking at the new item stats we have (like new chat, room, user, etc), there isn't an obvious spike, https://app.datadoghq.com/dash/760607/new-object-stats?live=true&page=0&is_auto=false&from_ts=1522789056558&to_ts=1522875456558&tile_size=m