This document shows how to safely upgrade the operator to a desired version while preserving the cluster's state and data whenever possible.
Note: Only applies to when upgrading from an etcd operator with version < v0.7.0.
First create a backup of your current cluster before starting the upgrade process. See the backup service guide on how to create a backup.
In the case of an upgrade failure you can restore your cluster to the previous state from the previous backup. See the spec examples on how to do that.
Note: if your cluster specifies either the backup policy or restore policy, then follow the migrate CR guide to update the cluster spec before upgrading the etcd-operator deployment.
Update the etcd-operator deployment:
Edit the existing etcd-operator deployment:
kubectl edit deployment/<your-operator>.
command field doesn't exist, add a new
command field to run
command: - etcd-operator
command field exists and
pv-provisioner flag is used, you must remove
v0.6.1+ the operator will no longer create a storage class specified by
--pv-provisioner by default. This behavior is set by the new flag
--create-storage-class which by default is
Note: If your cluster does not have the following backup policy then you can simply upgrade the operator to the
Backup policy that has
pv.storageClass is unset. For example:
spec: backup: backupIntervalInSecond: 30 maxBackups: 5 pv: storageClass: "" volumeSizeInMB: 512 storageType: PersistentVolume
So if your cluster has the above backup policy then do the following steps before upgrading the operator image to
Confirm the name of the storage class for a given cluster:
kubectl -n <namespace> get pvc -l=etcd_cluster=<cluster-name> -o yaml | grep storage-class
Edit your etcd cluster spec by changing the
spec.backup.pv.storageClass field to the name of the existing storage class from the previous step.
Wait for the the backup sidecar to be updated.
The v0.6.0 release removes operator S3 flag.
Note: if your cluster is not using operator S3 flag, then you just need recreate etcd-operator deployment with the
If your cluster is using operator S3 flag for backup and want to use S3 backup in v0.6.0, then you need to migrate your cluster to use cluster level backup.
Steps for migration:
Create a backup of you current data using the backup service guide.
Create a new AWS secret
<aws-secret> for cluster level backup:
$ kubectl -n <namespace> create secret generic <aws-secret> --from-file=$AWS_DIR/credentials --from-file=$AWS_DIR/config
Change the backup field in your existing cluster backup spec to enable cluster level backup:
backup: backupIntervalInSecond: 30 maxBackups: 5 storageType: "S3"
backup: backupIntervalInSecond: 30 maxBackups: 5 storageType: "S3" s3: s3Bucket: <your-s3-bucket> awsSecret: <aws-secret>
Apply the cluster backup spec:
$ kubectl -n <namespace> apply -f <your-cluster-deployment>.yaml
Update deployment spec to use
etcd-operator:v0.6.0 and remove the dependency on operator level S3 backup flags:
spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.5.2 command: - /usr/local/bin/etcd-operator - --backup-aws-secret=aws - --backup-aws-config=aws - --backup-s3-bucket=<your-s3-bucket>
spec: containers: - name: etcd-operator image: quay.io/coreos/etcd-operator:v0.6.0 command: - /usr/local/bin/etcd-operator
Apply the updated deployment spec:
$ kubectl -n <namespace> apply -f <your-etcd-operator-deployment>.yaml
0.4.x versions, please update to
Kubernetes cluster version must be 1.7+.
kubectl -n <namespace> delete cluster <cluster-name>
kubectl delete thirdpartyresource cluster.etcd.coreos.com
ClusterRolewith the new rules for CRD.
apiVersion: "etcd.database.coreos.com/v1beta2" kind: "EtcdCluster" metadata: name: <cluster-name> spec: size: <cluster-size> version: "3.1.8" backup: # The same backup spec used to save the backup of the previous cluster . . . . . . restore: backupClusterName: <previous-cluster-name> storageType: <storage-type-of-backup-spec>
The two points of interest in the above CR spec are:
kindfields have been changed as mentioned in the
spec.restorefield needs to be specified according your backup configuration. See spec examples guide on how to specify the
spec.restorefield for your particular backup configuration.
See the release notes of
v0.4.0 for noticeable changes: https://github.com/coreos/etcd-operator/releases/tag/v0.4.0
To upgrade to
v0.3.x the current operator verison must first be upgraded to
v0.2.6+, since versions <
v0.2.6 are not compatible with
Spec.Backup.MaxBackups < 0, previously it had no effect. Now it will get rejected. Please remove it.
Spec.Backup.MaxBackups == 0, previously it had no effect. Now it will create backup sidecar that allows unlimited backups. If you don't want backup sidecar, just don't set any backup policy.