Bootstrap embedded metadata store

[tillrohrmann]

When running an embedded metadata store, we need to bootstrap them with an initial configuration (peers of the Raft/OmniPaxos cluster). To avoid introducing a new id which users would have to configure independently of the NodeId and the node name, we want to reuse the NodeId of a node as its peer id. Since we need to write the NodesConfiguration to the metadata store to obtain our NodeId, we have a chicken-egg problem.

To solve this problem, we can introduce an explicit signal that tells one node to be the seed node of the Raft/OmniPaxos cluster. This node can create an initial NodesConfiguration and start the metadata store with itself as its only peer. Once this has happened, all other nodes can update the NodesConfiguration in the now running metadata store to obtain their node id. Later we can reconfigure the metadata store cluster to include other metadata store nodes as peers to increase its resilience.

Tasks

Preview Give feedback

1. Let MetadataStore use the node grpc server #2411
   +0
   
2. Add provision command to bootstrap metadata store and write initial NodesConfiguration #2409
   +0
   
3. Introduce --production profile which disables --allow-bootstrap #2410
   +0
4. Add support for reconfiguring embedded metadata store #2412
   +0
5. Add restatectl command to add metadata store peer members #2413
   +0
6. Add metadata store controller that reconfigures the metadata store nodes automatically #2414
   +0
7. Detect loss of metadata store disks #2415
   +0