/etc/origin/master/master-config.yaml.<timestamp> /var/lib/etcd/openshift-backup-pre-upgrade-<timestamp> /etc/origin/master/scheduler.json.<timestamp>
Following an OpenShift Container Platform upgrade, it may be desirable in extreme cases to downgrade your cluster to a previous version. The following sections outline the required steps for each system in a cluster to perform such a downgrade for the OpenShift Container Platform 3.9 to 3.7 downgrade path.
|
You can downgrade directly from 3.9 to 3.7, but you must restore from the etcd backup. If the upgrade failed at the 3.8 step, then the same downgrade procedures apply. |
|
These steps are currently only supported for RPM-based installations of OpenShift Container Platform and assumes downtime of the entire cluster. |
The Ansible playbook used during the upgrade process should have created a backup of the master-config.yaml file and the etcd data directory. Ensure these exist on your masters and etcd members:
/etc/origin/master/master-config.yaml.<timestamp> /var/lib/etcd/openshift-backup-pre-upgrade-<timestamp> /etc/origin/master/scheduler.json.<timestamp>
Also, back up the node-config.yaml file on each node (including masters, which have the node component on them) with a timestamp:
/etc/origin/node/node-config.yaml.<timestamp>
When using an external etcd cluster, the backup is created on all etcd members, though only one is required for the recovery process.
Keep a copy of the .rpmsave backups of the following files:
/etc/sysconfig/atomic-openshift-master-api /etc/sysconfig/atomic-openshift-master-controller /etc/etcd/etcd.conf
|
Restore from the first pre-upgrade backup taken on the day of the upgrade. |
On all masters, nodes, and etcd members (if using an external etcd cluster), ensure the relevant services are stopped.
# systemctl stop atomic-openshift-master-api atomic-openshift-master-controllers
On all master and node hosts:
# systemctl stop atomic-openshift-node
On any external etcd hosts:
# systemctl stop etcd
The *-excluder packages add entries to the exclude directive in the host’s /etc/yum.conf file when installed.
On all masters, nodes, and etcd members (if using a dedicated etcd cluster), remove the following packages:
# yum remove atomic-openshift \
atomic-openshift-clients \
atomic-openshift-node \
atomic-openshift-master-api \
atomic-openshift-master-controllers \
openvswitch \
atomic-openshift-sdn-ovs \
tuned-profiles-atomic-openshift-node\
atomic-openshift-excluder\
atomic-openshift-docker-excluder
If you are using external etcd, also remove the etcd package:
# yum remove etcd
OpenShift Container Platform 3.9 requires Docker 1.13, while OpenShift Container Platform 3.7 requires Docker 1.12.
On any host running Docker 1.13, downgrade to Docker 1.12 using the following steps:
Remove all local containers and images on the host. Any pods backed by a replication controller will be recreated.
|
The following commands are destructive and should be used with caution. |
Delete all containers:
# docker rm $(docker ps -a -q) -f
Delete all images:
# docker rmi $(docker images -q)
Use yum swap (instead of yum downgrade) to install Docker 1.12.6:
# yum swap docker-* docker-*1.12.6 -y # sed -i 's/--storage-opt dm.use_deferred_deletion=true//' /etc/sysconfig/docker-storage # systemctl restart docker
You should now have Docker 1.12.6 installed and running on the host. Verify with the following:
# docker version # systemctl status docker
Disable the OpenShift Container Platform 3.8 and 3.9 repositories, and re-enable the 3.7 repositories:
# subscription-manager repos \
--disable=rhel-7-server-ose-3.8-rpms \
--disable=rhel-7-server-ose-3.9-rpms \
--enable=rhel-7-server-ose-3.7-rpms
On each master, install the following packages:
# yum install atomic-openshift \
atomic-openshift-clients \
atomic-openshift-node \
atomic-openshift-master-api \
atomic-openshift-master-controllers \
openvswitch \
atomic-openshift-sdn-ovs \
tuned-profiles-atomic-openshift-node \
atomic-openshift-excluder \
atomic-openshift-docker-excluder
On each node, install the following packages:
# yum install atomic-openshift \
atomic-openshift-node \
openvswitch \
atomic-openshift-sdn-ovs \
tuned-profiles-atomic-openshift-node \
atomic-openshift-excluder \
atomic-openshift-docker-excluder
If using an external etcd cluster, install the following package on each etcd member:
# yum install etcd
See Backup and Restore.
See Backup and Restore.
To verify the downgrade, first check that all nodes are marked as Ready:
# oc get nodes NAME STATUS AGE master.example.com Ready,SchedulingDisabled 165d node1.example.com Ready 165d node2.example.com Ready 165d
Then, verify that you are running the expected versions of the docker-registry and router images, if deployed:
# oc get -n default dc/docker-registry -o json | grep \"image\"
"image": "openshift3/ose-docker-registry:v3.7.23",
# oc get -n default dc/router -o json | grep \"image\"
"image": "openshift3/ose-haproxy-router:v3.7.23",
You can use the diagnostics tool on the master to look for common issues and provide suggestions:
# oc adm diagnostics ... [Note] Summary of diagnostics execution: [Note] Completed with no errors or warnings seen.