Document and perform backup and restore of Gitter primary MongoDB.

This issue tracks process of documenting procedure for backing up and restoring Gitter primary database.

There is a 700GB mongo-prod-data-mongo-replica-02 volume that has its snapshot taken.

I'd assume this snapshot will need to be turned into a volume that gets attached to restored mongo instance.

  • understand and describe production mongodb replica set architecture and configuration
  • discuss the backup and restore procedure on this issue
  • create a MR with backup process and refer to it from the infrastructure README
  • follow the procedure in beta and possibly live-stream paring on the task
  • create an issue for doing the same for Gitter persistent Redis

Mongo replica set

We are running Mongo on 4 EC2 instances in production

  • mongo-replica-01, mongo-replica-02, mongo-replica-03, mongo-replica-arbiter

  • production replica set is named TroupeReplicaSet and configured in mongodb.conf.j2

  • how do the replicas get added? Tutorial says that rs.add() should be called but I can't find it in our infrastructure setup.

Details about production replica set

TroupeReplicaSet:PRIMARY> rs.status()

{
	"set" : "TroupeReplicaSet",
	"date" : ISODate("2019-06-24T12:42:21.065Z"),
	"myState" : 1,
	"term" : NumberLong(-1),
	"heartbeatIntervalMillis" : NumberLong(2000),
	"members" : [
		{
			"_id" : 2,
			"name" : "mongo-replica-arbiter:27017",
			"health" : 1,
			"state" : 7,
			"stateStr" : "ARBITER",
			"uptime" : 11678097,
			"lastHeartbeat" : ISODate("2019-06-24T12:42:19.183Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-24T12:42:20.490Z"),
			"pingMs" : NumberLong(1),
			"configVersion" : 45
		},
		{
			"_id" : 9,
			"name" : "mongo-replica-01:27017",
			"health" : 1,
			"state" : 1,
			"stateStr" : "PRIMARY",
			"uptime" : 86150348,
			"optime" : Timestamp(1561380141, 3),
			"optimeDate" : ISODate("2019-06-24T12:42:21Z"),
			"electionTime" : Timestamp(1475230014, 1),
			"electionDate" : ISODate("2016-09-30T10:06:54Z"),
			"configVersion" : 45,
			"self" : true
		},
		{
			"_id" : 10,
			"name" : "mongo-replica-02:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 7010548,
			"optime" : Timestamp(1561380139, 3),
			"optimeDate" : ISODate("2019-06-24T12:42:19Z"),
			"lastHeartbeat" : ISODate("2019-06-24T12:42:19.076Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-24T12:42:19.710Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "mongo-replica-03:27017",
			"configVersion" : 45
		},
		{
			"_id" : 11,
			"name" : "mongo-replica-03:27017",
			"health" : 1,
			"state" : 2,
			"stateStr" : "SECONDARY",
			"uptime" : 10329911,
			"optime" : Timestamp(1561380140, 5),
			"optimeDate" : ISODate("2019-06-24T12:42:20Z"),
			"lastHeartbeat" : ISODate("2019-06-24T12:42:20.271Z"),
			"lastHeartbeatRecv" : ISODate("2019-06-24T12:42:20.599Z"),
			"pingMs" : NumberLong(0),
			"syncingTo" : "mongo-replica-01:27017",
			"configVersion" : 45
		}
	],
	"ok" : 1
}

Mongo backup process

Mongo restore procedure

Edited by Tomas Vik