thanos-compact not compacting on thanos-compact-01-inf-gprd

Failed around 2020-10-28 01:27 UTC according to https://prometheus.gprd.gitlab.net/graph?g0.range_input=1h&g0.end_input=2020-10-28%2002%3A10&g0.expr=thanos_compactor_halted%20%3D%3D%201&g0.tab=0 which correlates with the log:

Oct 28 01:26:58 thanos-compact-01-inf-gprd thanos[29405]: {"caller":"compact.go:377","err":"compaction: group 0@16686929188435006227: compact blocks [/opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EJP50AZ3P1EN2NWMFPMC5BA7 /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EJQHMW5K9VP7FWW7VD3M9FQB /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EJWQJQ0M2F33JV47CEFKFK5N /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EK1XNFBHZ0MB86HBEZ9NFSVV /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EK711SSHZHPDH5V274M8ESAY /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01EKC6VXQ4A3ZANP0E30KY5AWM /opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01ENA7V7YW31BVM1QFRJ57TDVX]: \"/opt/prometheus/thanos/compact-data/compact/0@16686929188435006227/01ENNGZMAS2N3JEDBD2DCA7DRH.tmp-for-creation/index\" exceeding max size of 64GiB","level":"error","msg":"critical error detected; halting","ts":"2020-10-28T01:26:58.578272103Z"}

Edited by Craig Miskell