2019-09-26: InfluxDB nodes were not using the persistent data disk
InfluxDB nodes were not using the persistent data disk attached to them, instead, they were using the boot disk. This seems like an oversight from the time these nodes were provisioned, as their Terraform configuration should've specified /var/lib/influxdb as the persistent disk path.
Since we only have 4 of these nodes, a manual fix was applied:
- Chef configuration was updated to point to the persistent disk
-
chef-clientwas stopped on all nodes - Node by node,
influxdwas stopped, data dir was moved to the persistent disk, thenchef-clientwas executed