Nach einem unsanften Herunterfahren unseres Kubernetes-Clusters funktionierte der API-Server nicht mehr. Nach einigen Untersuchungen fand ich in jedem etcd-Mitglied auf jedem Knoten die folgenden Fehler.
Wie kann ich den Cluster wiederherstellen?
Meister1
2021-01-12 13:34:54.273559 I | etcdserver: recovered store from snapshot at index 143742270
2021-01-12 13:34:54.281853 I | mvcc: restore compact to 127098354
2021-01-12 13:34:54.310003 C | mvcc: store.keyindex: put with unexpected smaller revision [{127097381 0} / {127099854 0}]
panic: store.keyindex: put with unexpected smaller revision [{127097381 0} / {127099854 0}]
# ... stack trace ...
Master2
panic: freepages: failed to get all reachable pages (page 3630520571184623672: out of bounds: 11503)
# ... stack trace ...
Master3
2021-01-13 12:10:35.428458 I | etcdserver: recovered store from snapshot at index 143735303
2021-01-13 12:10:35.437350 I | mvcc: restore compact to 127098354
2021-01-13 12:10:35.481940 C | mvcc: store.keyindex: put with unexpected smaller revision [{127097229 0} / {127099849 0}]
panic: store.keyindex: put with unexpected smaller revision [{127097229 0} / {127099849 0}]
# ... stack trace ...