# subscription-manager refresh
If you installed using the advanced installation and the inventory file that was used is available, you can use upgrade playbooks to automate the OpenShift cluster upgrade process.
The OpenShift Container Platform 3.9 release includes a merge of features and fixes from Kubernetes 1.8 and 1.9. As a result, the upgrade process from OpenShift Container Platform 3.7 completes with the cluster fully upgraded to OpenShift Container Platform 3.9, seemingly "skipping" the 3.8 release. Technically, the OpenShift Container Platform 3.7 cluster is first upgraded to 3.8-versioned packages, and then the process immediately continues upgrading to OpenShift Container Platform 3.9 automatically. Your cluster should only remain at 3.8-versioned packages for as long as it takes to successfully complete the upgrade to OpenShift Container Platform 3.9.
|
As of OpenShift Container Platform 3.9, the quick installation method is deprecated. In a future release, it will be removed completely. In addition, using the quick installer to upgrade from version 3.7 to 3.9 is not supported. |
The automated 3.7 to 3.9 control plane upgrade performs the following steps for you:
A backup of all etcd data is taken for recovery purposes.
The API and controllers are updated from 3.7 to 3.8.
Internal data structures are updated to 3.8.
A second backup of all etcd data is taken for recovery purposes.
The API and controllers are updated from 3.8 to 3.9.
Internal data structures are updated to 3.9.
The default router, if one exists, is updated from 3.7 to 3.9.
The default registry, if one exists, is updated from 3.7 to 3.9.
The default image streams and InstantApp templates are updated.
The automated 3.7 to 3.9 node upgrade performs a rolling update of nodes, which:
Marks a subset of nodes unschedulable and drains them of pods.
Updates node components from 3.7 to 3.9 (including openvswitch and container runtime).
Returns those nodes to service.
|
Automated upgrade playbooks are run via Ansible directly using the
ansible-playbook command with an inventory file, similar to the advanced
installation method. The same v3_9 upgrade playbooks can be used for either
of the following scenarios:
Upgrading existing OpenShift Container Platform 3.7 clusters to 3.9
Upgrading existing OpenShift Container Platform 3.9 clusters to the latest asynchronous errata updates
|
Before upgrading your cluster to OpenShift Container Platform 3.9, the cluster must be already upgraded to the latest asynchronous release of version 3.7. If your cluster is at a version earlier than 3.7, you must first upgrade incrementally (e.g., 3.5 to 3.6, then 3.6 to 3.7). |
|
Before attempting the upgrade, follow the guidance in Environment health checks to verify the cluster’s health. This will confirm that nodes are in the Ready state, running the expected starting version, and will ensure that there are no diagnostic errors or warnings. |
To prepare for an automated upgrade:
Pull the latest subscription data from RHSM:
# subscription-manager refresh
If you are upgrading from OpenShift Container Platform 3.7 to 3.9:
Manually disable the 3.7 repository and enable both the 3.8 and 3.9 repositories on each master and node host. You must also enable the rhel-7-server-ansible-2.4-rpms repository, which is a new requirement starting with OpenShift Container Platform 3.9:
# subscription-manager repos --disable="rhel-7-server-ose-3.7-rpms" \
--enable="rhel-7-server-ose-3.8-rpms" \
--enable="rhel-7-server-ose-3.9-rpms" \
--enable="rhel-7-server-extras-rpms" \
--enable="rhel-7-server-ansible-2.4-rpms" \
--enable="rhel-7-fast-datapath-rpms"
# yum clean all
In previous versions of OpenShift Container Platform, master hosts were marked unschedulable by default by the installer, meaning that no pods could be placed on the hosts. Starting with OpenShift Container Platform 3.9, however, masters must be marked schedulable; this is done automatically during upgrade.
As a result of this requirement, the default node selector must also now be set
by default; this is also done automatically during upgrade if it is unset,
otherwise it is left as is. For either scenario, masters are automatically
labeled with the master node role, and non-master, non-infrastructure nodes
are labeled with the compute node role.
See the OpenShift Container Platform 3.9 Release Notes on the these notable technical changes and ensure that you understand their impact on your cluster:
You must disable swap memory in your cluster before upgrading to OpenShift Container Platform
3.9, otherwise the upgrade will fail. Whether swap memory was enabled using
openshift_disable_swap=false in your Ansible inventory file or enabled
manually per host, see
Disabling Swap Memory
in the Cluster Administration guide to disable it on each host.
For any upgrade path, always ensure that you have the latest version of the atomic-openshift-utils package on each RHEL 7 system, which also updates the openshift-ansible-* packages:
# yum update atomic-openshift-utils
After satisfying these steps, you can review the following sections for more information on how the upgrade process works and make decisions on additional upgrade customization options if you so choose. When you are prepared to run the upgrade, you can continue to Upgrading to the Latest OpenShift Container Platform 3.9 Release.
An OpenShift Container Platform cluster can be upgraded in one or more phases. You can choose whether to upgrade all hosts in one phase by running a single Ansible playbook, or upgrade the control plane (master components) and nodes in multiple phases using separate playbooks.
|
Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.9 Release. |
When upgrading in separate phases, the control plane phase includes upgrading:
master components
node services running on masters
Docker running on masters
Docker running on any stand-alone etcd hosts
When upgrading only the nodes, the control plane must already be upgraded. The node phase includes upgrading:
node services running on stand-alone nodes
Docker running on stand-alone nodes
|
Nodes running master components are not included during the node upgrade phase, even though they have node services and Docker running on them. Instead, they are upgraded as part of the control plane upgrade phase. This ensures node services and Docker on masters are not upgraded twice (once during the control plane phase and again during the node phase). |
Whether upgrading in a single or multiple phases, you can customize how the node
portion of the upgrade progresses by passing certain Ansible variables to an
upgrade playbook using the -e option.
|
Instructions on the full upgrade process and when to call these playbooks are described in Upgrading to the Latest OpenShift Container Platform 3.7 Release. |
The openshift_upgrade_nodes_serial variable can be set to an integer or
percentage to control how many node hosts are upgraded at the same time. The
default is 1, upgrading nodes one at a time.
For example, to upgrade 20 percent of the total number of detected nodes at a time:
$ ansible-playbook -i <path/to/inventory/file> \
</path/to/upgrade/playbook> \
-e openshift_upgrade_nodes_serial="20%"
The openshift_upgrade_nodes_label variable allows you to specify that only
nodes with a certain label are upgraded. This can also be combined with the
openshift_upgrade_nodes_serial variable.
For example, to only upgrade nodes in the group1 region, two at a time:
$ ansible-playbook -i <path/to/inventory/file> \
</path/to/upgrade/playbook> \
-e openshift_upgrade_nodes_serial="2" \
-e openshift_upgrade_nodes_label="region=group1"
|
See Managing Nodes for more on node labels. |
The openshift_upgrade_nodes_max_fail_percentage variable allows you to specify
how many nodes may fail in each batch. The percentage of failure must exceed
your value before the playbook aborts the upgrade.
The openshift_upgrade_nodes_drain_timeout variable allows you to specify the
length of time to wait before giving up.
In this example, 10 nodes are upgraded at a time, the upgrade will abort if more than 20 percent of the nodes fail, and there is a 600-second wait to drain the node:
$ ansible-playbook -i <path/to/inventory/file> \
</path/to/upgrade/playbook> \
-e openshift_upgrade_nodes_serial=10 \
-e openshift_upgrade_nodes_max_fail_percentage=20 \
-e openshift_upgrade_nodes_drain_timeout=600
When upgrading OpenShift Container Platform, you can execute custom tasks during specific operations through a system called hooks. Hooks allow cluster administrators to provide files defining tasks to execute before and/or after specific areas during upgrades. This can be very helpful to validate or modify custom infrastructure when upgrading OpenShift Container Platform.
It is important to remember that when a hook fails, the operation fails. This means a good hook can run multiple times and provide the same results. A great hook is idempotent.
Hooks have no defined or versioned interface. They can use internal openshift-ansible variables, but there is no guarantee these will remain in future releases. In the future, hooks may be versioned, giving you advance warning that your hook needs to be updated to work with the latest openshift-ansible.
Hooks have no error handling, so an error in a hook will halt the upgrade process. The problem will need to be addressed and the upgrade re-run.
Hooks are defined in the hosts inventory file under the OSEv3:vars
section.
Each hook must point to a YAML file which defines Ansible tasks. This file will be used as an include, meaning that the file cannot be a playbook, but a set of tasks. Best practice suggests using absolute paths to the hook file to avoid any ambiguity.
[OSEv3:vars]
openshift_master_upgrade_pre_hook=/usr/share/custom/pre_master.yml
openshift_master_upgrade_hook=/usr/share/custom/master.yml
openshift_master_upgrade_post_hook=/usr/share/custom/post_master.yml---
# Trivial example forcing an operator to ack the start of an upgrade
# file=/usr/share/custom/pre_master.yml
- name: note the start of a master upgrade
debug:
msg: "Master upgrade of {{ inventory_hostname }} is about to start"
- name: require an operator agree to start an upgrade
pause:
prompt: "Hit enter to start the master upgrade"openshift_master_upgrade_pre_hookRuns before each master is upgraded.
This hook runs against each master in serial.
If a task must run against a different host, said task must use
delegate_to
or local_action.
openshift_master_upgrade_hookRuns after each master is upgraded, but before its service or system restart.
This hook runs against each master in serial.
If a task must run against a different host, said task must use
delegate_to
or local_action.
openshift_master_upgrade_post_hookRuns after each master is upgraded and has had its service or system restart.
This hook runs against each master in serial.
If a task must run against a different host, said task must use
delegate_to
or local_action.
To upgrade an existing OpenShift Container Platform 3.7 or 3.9 cluster to the latest 3.9 release:
Satisfy the steps in Preparing for an Automated Upgrade to ensure you are using the latest upgrade playbooks.
Ensure the openshift_deployment_type parameter in your inventory file is set
to openshift-enterprise.
If you have multiple masters configured and want to enable rolling, full system
restarts of the hosts, you can set the openshift_rolling_restart_mode
parameter in your inventory file to system. Otherwise, the default value
services performs rolling service restarts on HA masters, but does not reboot
the systems. See
Configuring
Cluster Variables for details.
At this point, you can choose to run the upgrade in a single or multiple phases. See Upgrading the Control Plane and Nodes in Separate Phases for more details which components are upgraded in each phase.
If your inventory file is located somewhere other than the default
/etc/ansible/hosts, add the -i flag to specify its location. If you
previously used the atomic-openshift-installer command to run your
installation, you can check ~/.config/openshift/hosts for the last inventory
file that was used, if needed.
Option A) Upgrade control plane and nodes in a single phase.
Run the upgrade.yml playbook to upgrade the cluster in a single phase using one playbook; the control plane is still upgraded first, then nodes in-place:
# ansible-playbook -i </path/to/inventory/file> \
/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml
Option B) Upgrade the control plane and nodes in separate phases.
To upgrade only the control plane, run the upgrade_control_plane.yaml playbook:
# ansible-playbook -i </path/to/inventory/file> \
/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml
To upgrade only the nodes, run the upgrade_nodes.yaml playbook:
# ansible-playbook -i </path/to/inventory/file> \
/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_nodes.yml \
[-e <customized_node_upgrade_variables>] (1)
| 1 | See Customizing Node Upgrades for any desired
<customized_node_upgrade_variables>. |
If you are upgrading the nodes in groups as described in Customizing Node Upgrades, continue invoking the upgrade_nodes.yml playbook until all nodes have been successfully upgraded.
After all master and node upgrades have completed, reboot all hosts. After rebooting, if there are no additional features enabled, you can verify the upgrade. Otherwise, the next step depends on what additional features you have previously enabled.
| Feature | Next Step |
|---|---|
Aggregated Logging |
|
Cluster Metrics |
Starting with OpenShift Container Platform 3.9, the web console is deployed as a separate
component rather than as part of the master. The console will be installed by
default as part of an automated upgrade unless you set the
openshift_web_console_install variable to false.
Alternatively, you can install the web console separately if needed, for example as part of a manual upgrade:
See the following section in the Advanced Installation topic and update your inventory file accordingly.
Run the following playbook:
# ansible-playbook -i </path/to/inventory/file> \
/usr/share/ansible/openshift-ansible/playbooks/openshift-web-console/config.yml
|
The way the web console loads extensions has changed in OpenShift Container Platform 3.9. If you are upgrading from an earlier version, you must host your extension scripts and stylesheets at a URL instead of on the master file system. See the Loading Extension Scripts and Stylesheets topic for details. |
To upgrade an existing EFK logging stack deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-logging/config.yml Ansible playbook. This is the playbook to use if you were deploying logging for the first time on an existing cluster, but is also used to upgrade existing logging deployments.
If you have not already done so, see
Specifying Logging Ansible Variables in the
Aggregating Container Logs topic and update your Ansible inventory file to at least set the
following required variable within the [OSEv3:vars] section:
[OSEv3:vars] openshift_logging_install_logging=true (1) openshift_logging_image_version=<tag> (2)
| 1 | Enables the ability to upgrade the logging stack. |
| 2 | Replace <tag> with v3.9.14 for the latest version. |
Add any other openshift_logging_* variables that you want to specify to
override the defaults, as described in
Specifying Logging Ansible Variables.
When you have finished updating your inventory file, follow the instructions in Deploying the EFK Stack to run the openshift-logging/config.yml playbook and complete the logging deployment upgrade.
To upgrade an existing cluster metrics deployment, you must use the provided /usr/share/ansible/openshift-ansible/playbooks/openshift-metrics/config.yml Ansible playbook. This is the playbook to use if you were deploying metrics for the first time on an existing cluster, but is also used to upgrade existing metrics deployments.
If you have not already done so, see
Specifying
Metrics Ansible Variables in the
Enabling
Cluster Metrics topic and update your Ansible inventory file to at least set
the following required variables within the [OSEv3:vars] section:
[OSEv3:vars] openshift_metrics_install_metrics=true (1) openshift_metrics_image_version=<tag> (2) openshift_metrics_hawkular_hostname=<fqdn> (3) openshift_metrics_cassandra_storage_type=(emptydir|pv|dynamic) (4)
| 1 | Enables the ability to upgrade the metrics deployment. |
| 2 | Replace <tag> with v3.9.14 for the latest version. |
| 3 | Used for the Hawkular Metrics route. Should correspond to a fully qualified domain name. |
| 4 | Choose a type that is consistent with the previous deployment. |
Add any other openshift_metrics_* variables that you want to specify to
override the defaults, as described in
Specifying
Metrics Ansible Variables.
When you have finished updating your inventory file, follow the instructions in Deploying the Metrics Deployment to run the openshift-metrics/config.yml playbook and complete the metrics deployment upgrade.
When upgrading OpenShift Container Platform, you must upgrade the set of nodes where GlusterFS pods are running.
Special consideration must be taken when upgrading these nodes, as drain and
unschedule will not terminate and evacuate the GlusterFS pods because they are
running as part of a daemonset.
There is also the potential for someone to run an upgrade on multiple nodes at the same time, which would lead to data availability issues if more than one was hosting GlusterFS pods.
Even if a serial upgrade is running, there is no guarantee sufficient time will be given for GlusterFS to complete all of its healing operations before GlusterFS on the next node is terminated. This could leave the cluster in a bad or unknown state. Therefore, the following procedure is recommended.
Upgrade the control plane (the master nodes and etcd nodes).
Upgrade standard infra nodes (router, registry, logging, and metrics).
|
If any of the nodes in those groups are running GlusterFS, perform step 4 of
this procedure at the same time. GlusterFS nodes must be upgraded along with
other nodes in their class ( |
Upgrade standard nodes running application containers.
|
If any of the nodes in those groups are running GlusterFS, perform step 4 of
this procedure at the same time. GlusterFS nodes must be upgraded along with
other nodes in their class ( |
Upgrade the OpenShift Container Platform nodes running GlusterFS one at a time.
Run oc get daemonset to verify the label found under NODE-SELECTOR. The
default value is storagenode=glusterfs.
Remove the daemonset label from the node:
$ oc label node <node_name> <daemonset_label>-
This will cause the GlusterFS pod to terminate on that node.
Add an additional label (for example, type=upgrade) to the node you want to upgrade.
To run the upgrade playbook on the single node where you terminated GlusterFS,
use -e openshift_upgrade_nodes_label="type=upgrade".
When the upgrade completes, relabel the node with the daemonset selector:
$ oc label node <node_name> <daemonset_label>
Wait for the GlusterFS pod to respawn and appear.
oc rsh into the pod and verify all volumes are healed:
$ oc rsh <GlusterFS_pod_name> $ for vol in `gluster volume list`; do gluster volume heal $vol info; done
Ensure all of the volumes are healed and there are no outstanding tasks. The
heal info command lists all pending entries for a given volume’s heal process.
A volume is considered healed when Number of entries for that volume is 0.
Remove the upgrade label (for example, type=upgrade) and go to the next
GlusterFS node.
Because the default gcePD storage provider uses an RWO (Read-Write Only) access mode, you cannot perform a rolling upgrade on the registry or scale the registry to multiple pods. Therefore, when upgrading OpenShift Container Platform, you must specify the following environment variables in your Ansible inventory file:
[OSEv3:vars] openshift_hosted_registry_storage_provider=gcs openshift_hosted_registry_storage_gcs_bucket=bucket01 openshift_hosted_registry_storage_gcs_keyfile=test.key openshift_hosted_registry_storage_gcs_rootdirectory=/registry
Ensure that the:
cluster is healthy,
services (master, node, and etcd) are running well,
the OpenShift Container Platform, docker-registry, and router versions are correct,
the original applications are still available and the new application can be created, and
running oc adm diagnostics produces no errors.
To verify the upgrade:
Check that all nodes are marked as Ready:
# oc get nodes NAME STATUS ROLES AGE VERSION master.example.com Ready master 7h v1.9.1+a0ce1bc657 node1.example.com Ready compute 7h v1.9.1+a0ce1bc657 node2.example.com Ready compute 7h v1.9.1+a0ce1bc657
Verify that you are running the expected versions of the docker-registry
and router images, if deployed.
Replace <tag> with v3.9.14 for the latest version.
# oc get -n default dc/docker-registry -o json | grep \"image\"
"image": "openshift3/ose-docker-registry:<tag>",
# oc get -n default dc/router -o json | grep \"image\"
"image": "openshift3/ose-haproxy-router:<tag>",
Use the diagnostics tool on the master to look for common issues:
# oc adm diagnostics ... [Note] Summary of diagnostics execution: [Note] Completed with no errors or warnings seen.