Irmin error with mainnet rolling filesystem on v10.3 from v10-release node
We copied a mainnet rolling node filesystem running v10-release to a node running v10.3. We use an intermittent v11.0-rc2 container to "initialize" the filesystem in case there are any head errors. Since pretty much EOD 11-13 our mainnet rolling filesystems have been producing the following errors on our mainnet v10.3 node.
│ + /usr/local/bin/tezos-node run --bootstrap-threshold 0 --config-file /etc/tezos/config.json │
│ Nov 15 17:25:29.506 - node.config.validation: the node configuration has been successfully validated. │
│ Nov 15 17:25:29.506 - node.main: read identity file (peer_id = idqno83x5sKExS1Ge5QLSDmbKcRHjP) │
│ Nov 15 17:25:29.506 - node.main: starting the Tezos node (chain = TEZOS_MAINNET) │
│ Nov 15 17:25:29.506 - node.main: disabled local peer discovery │
│ Nov 15 17:25:29.532 - node: shell-node initialization: bootstrapping │
│ Nov 15 17:25:29.538 - node: shell-node initialization: p2p_maintain_started │
│ Nov 15 17:25:29.538 - external_block_validator: initialized │
│ Nov 15 17:25:29.778 - external_block_validator: block validator process started with pid 67 │
│ tezos-node: Error: │
│ Irmin_pack__Pack_store.Maker(V)(Index)(K).Make_without_close_checks(Val).Invalid_read │
│ │
│ + sleep 120 │
│ + sleep 3600Here are LZ4 tar archives of /var/tezos/node that will reproduce this error. (4.5 GB each)
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1859962.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1860136.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1860362.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1860549.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1860741.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1860908.lz4
https://tezos-tarballs.s3.us-east-2.amazonaws.com/tezos-mainnet-rolling-tarball-1861240.lz4