Cluster Membership Reconciliation
Given a desired size S, we have two membership states:
- running pods P in k8s cluster
- membership M in operator knowledge
Reconciliation is the process to make these two states consistent with the desired size S.
For each reconciling cycle, we get P from k8s API. Comparing M and P, we have the following steps:
- Remove all pods from set P that does not belong to set M
- P’ consist of remaining pods of P
- If P’ = M, the current state matches the membership state. GOTO Resize.
- If len(P’) < len(M)/2 + 1, quorum lost. Go to recovery process (TODO).
- Remove one member that in M but does not in P's. GOTO Resize.
Given a desired size S, membership M and current running pods P:
- If len(P) != len(M), then END.
- If S = len(M), then END.
- If S > len(M), add one member. END.
- If S < len(M), remove one member. END