OpenShift Container Platform 3.9 Release Notes

non-master, non-dedicated infrastructure nodes hosts (by default, this means nodes with a region=infra label) are labeled with node-role.kubernetes.io/compute=true, which assigns the compute node role.

master nodes are labeled with node-role.kubernetes.io/master=true, which assigns the master node role.

new

new-basicauth

new-dockercfg

new-sshauth

template_service_broker_prefix="registry.example.com/openshift3/"

template_service_broker_image_name="ose-template-service-broker"

template_service_broker_prefix="registry.example.com/openshift3/ose-"

template_service_broker_image_name="template-service-broker"

Previously, builds selected the secret to be used for pushing the output image at the time they were started. When a build started before the default service account secrets for a project were created, the build may not have found a suitable secret for pushing the image, resulting in the build failing when it went to push the image. With this fix, the build is held until the default service account secrets exist, ensuring that if the default secret is suitable for pushing the image, it can and will be used. As a result, initial builds in a newly created project are no longer at risk of failing if the build is created before the default secrets are populated. (BZ#1333030)

The systemd units for masters changed without the diagnostics being updated. This caused the diagnostics to silently check for master systemd units that did not exist, and problems were not reported. With this fix, diagnostics check for correct master unit names and problems with master systemd units and logs may be found. (BZ#1378883)

If a container shares namespace with another container, then they would share the namespace path. If you run the exec command in the first container, it only reads the namespace paths stored in the file and joins those namespaces. So, if the second container has already been stopped, the exec command in the first container will fail. As a result, this fix saves namespace paths no matter if containers share namespaces. (BZ#1510573)

Docker has a known "zombie process" phenomenon that impacted the OpenShift Jenkins image, causing operating system-level resources to be exhausted as these “zombie processes” accumulated. With this fix, the OpenShift Jenkins image now leverages one of the Docker image init implementations to launch Jenkins, monitor, and handle any “zombie child processes”. As a result, “zombie processes” no longer accumulate. (BZ#1528548)

Due to a fault in the scheduler implementation, the ScheduledImageImportMinimumIntervalSeconds setting was not correctly observed, causing OpenShift Container Platform to attempt to import scheduled images at the wrong intervals. This is now resolved. (BZ#1543446)

Previously, OpenShift would erroneously re-import all tags on an image stream, regardless if marked as scheduled or not, if any tag on the image stream was marked as scheduled. This behavior is now resolved. (BZ#1515060)

The signature importer tried to import signatures from the internal registry without credentials, causing the registry to check if the anonymous user could get signatures using SAR requests. With this bug fix, the signature importer skips the internal registry because the internal registry and the signature importer work with the same storage, resulting in no SAR requests. (BZ#1543122)

There was no check of the number of components in the path, causing the data to be placed in the storage but not be written to the database. With this bug fix, an early check of the path was added. (BZ#1528613)

The Kubernetes service IP address was not added to no_proxy list for the docker-registry during installation. As a result, internal registry requests would be forced to use the proxy, preventing logins and pushes to the internal registry. The installer was changed to add the Kubernetes service IP to the no_proxy list. (BZ#1504464)

The installer was pulling the incorrect efs-provisioner image, which caused the installation of the provisioner pod to fail to deploy. The installer was changed to pull the correct image. (BZ#1523534)

When installing OpenShift Container Platform with a custom registry, the installer was using the default registry. The registry console default image is now defined as a fully qualified image registry.access.redhat.com/openshift3/registry-console which means that when a custom registry is specified via oreg_url and image streams are modified to use that custom registry the registry console will also utilize the custom registry. (BZ#1523638)

Running the redeploy-etcd-ca.yml playbook did not update the ca.crt used by etcd system container. The code was changed so that the playbook properly updates the the etcd ca.crt in /etc/etcd/ca.crt as expected. (BZ#1466216)

Following a successful deployment of CNS/CRS with glusterblock, OpenShift Container Platform logging and metrics can be deployed using glusterblock as their backend storage for fault-tolerant, distributed persistent storage. (BZ#1480835)

When upgrading from 3.6 to 3.7, the user wanted the Hawkular OpenShift Agent pods deactivated. But, after upgrade, the HOSA pods are still being deployed. A new playbook, uninstall_hosa.yaml, has been created to remove HOSA from a OpenShift Container Platform cluster when openshift_metrics_install_hawkular_agent=false in the Ansible inventory file. (BZ#1497408)

Because registry credentials for the broker were stored in a ConfigMap, sensitive credentials could be exposed in plain text. A secret is now created to store the credentials Registry credentials are no longer visible in plaintext. (BZ#1509082)

Because of incorrect naming, the uninstall playbook did not remove the tuned-profiles-atomic-openshift-node package. The playbook is now corrected and the package is removed upon uninstallation of OpenShift Container Platform. (BZ#1509129)

When running the installer with the openshift_hosted_registry_storage_volume_size parameter configured with Jnja code, the installation failed during persistent volume creation. The code is now fixed to properly interpret the Jinja code. (BZ#1518386)

During disconnected installations, the service catalog was attempting to pull down images from the configured registry. This caused the installation to fail as the registry is not available during a disconnected installation. The imagePullPolicy in the installer was changed to ifNotPresent. If the image is present, the service catalog will not attempt to pull it again, and the disconnected installation of the service catalog will proceed. (BZ#1524805)

When provisioning hosts with an SSH proxy configured, the masters would never appear marked as up. With this bug fix, the task is changed to use an Ansible module that respects SSH proxy configuration. As a result, Ansible is able to connect to the hosts and they are marked as up. (BZ#1541946)

In an HTTPS environment, the service catalog installation was failing because the playbook attempted to contact the API server using cURL without the --noproxy option specified. The command in the playbook was changed to include --noproxy and the installer performs as expected. (BZ#1544645)

Previously, the storage type for Elasticsearch data centers was not preserved when upgrading/rerunning. This caused the existing storage type to be overwritten. This bug fix preserves the storage type as the default (using an inventory variable if specified). (BZ#1496758)

Previously, the docker daemon was incorrectly restarted when redeploying node certificates. This caused unnecessary downtime in nodes since atomic-openshift-node was the only component loading the kubeconfig. This bug fix adds a flag to check if a new Certificate Authority (CA) is being deployed. If not, then restarting Docker is skipped. (BZ#1537726)

Previously, the docker_image_availability check did not take into account variables that override specific container images used for containerized components. This caused the check to incorrectly report failures when looking for the default images when the overridden images were actually available. As a result of this bug fix, the check should accurately report whether the necessary images are available. (BZ#1538806)

When determining if a persistent volume claim (PVC) should be created for Elasticsearch, we used a legacy variable, which did not correctly evaluate if a PVC was necessary when creating a Network File System (NFS)-backed persistent volume (PV). This bug fix correctly evaluates if a PVC is necessary for the deployment configuration. (BZ#1538995)

Previously, when configuring the registry for Azure Blob storage, the realm of core.windows.net was specified by default. This bug fix allows you to change openshift_hosted_registry_storage_azure_blob_realm to the value that you want to use. (BZ#1491100)

A new playbook has been introduced that uninstalls an existing GlusterFS deployment. This playbook removes all existing resources, including pods and services. This playbook also, optionally, removes all data and configuration from the hosts that were running GlusterFS pods. (BZ#1497038)

Previously, the OpenShift Container Platform logging system did not support CRI-O. This bug fix added a parser for CRI-O formatted logs. As a result, both system and container logs can be collected. (BZ#1517605)

When redeploying logging, we previously attempted to maintain any changes that were made to the ConfigMaps post-installation. It was difficult to let users specify the contents of a ConfigMap file while still needing the ability to provide the configurations required for the different Elasticsearch, Fluentd, and Kibana (EFK) stack components. This bug fix created a patch based on changes made post-deployment and applies that patch to the files provided by the installer. (BZ#1519619)

The Kibana page previously displayed OPENSHIFT ORIGIN in the upper left-hand corner of the OpenShift Container Platform web console. This bug fix replaces the Origin header image with the OpenShift Container Platform header image. As a result, the Kibana page now displays the desired header. (BZ#1546311)

Both the OpenShift Container Platform DeploymentConfig and Kubernetes extensions/v1beta1 Deployment resources were labeled with deployment on the web console overview, so you could not differentiate the resources. DeploymentConfig resources on the Overview page are now labelled with DeploymentConfig. (BZ#1488380)

The web console’s pod status filter did not correctly display pod init status when an error prevented the pod from initializing, including and init status of error. If a pod has an Init:Error status, the pod status correctly displays Init Error instead of Pod Initializing. (BZ#1512473)

Previously, switching tabs in the web console page for a pipeline build configuration caused some content on the page to no longer be visible while the page reloaded. Switching tabs no longer reloads the entire page, and content is correctly displayed. (BZ#1527346)

By default, an old version of the builder image was shown when you added a builder to a project and selected by default during builder configuration. This gave the wrong impression that your only choice was an old version of a language or framework. The version number is no longer shown in the wizard title, and the newest available version is selected by default. (BZ#1542669)

If you used some browsers, you could not consistently use the right click menu to copy and paste text from internal editors that used the ACE editor library, including the YAML, Jenkinsfile, and Dockerfile editors. This update uses a newer version of the ACE editor library, so the right click menu options work throughout the console. (BZ#1463617)

Previously, browsers would use the default behavior for the Referrer-Policy because Referrer-Policy header was not sent by the console. Now the console correctly sends the Referrer-Policy header, which is set to strict-origin-when-cross-origin, and browsers that listen to the Referrer-Policy header follow the strict-origin-when-cross-origin policy for the web console. (BZ#1504571)

Previously, users with read access to the project saw webhook secret values because they were stored as strings in the build. These users could use these values to trigger builds even though they had only read access to the project. Now webhook secrets are defined as secret objects in the build instead of strings. Users with read only access to the project cannot see the secret values or use them to trigger builds by using the webhook. (BZ#1504819)

Previously, adding the same persistent volume claim more than once to a deployment in the web console caused pods for that deployment to fail. The web console incorrectly created a new volume when it added the second PVC to the deployment instead of reusing the existing volume from the pod template spec. Now, the web console reuses the existing volume if the same PVC is listed more than once. This behavior lets you add the same PVC with different mount paths and subpaths as needed. (BZ#1527689)

Previously, it was not clear enough that you can not select an Image Name from the Deploy Image window if you are also creating a new project. The help text that explains that you can only set an Image Name for existing projects is easier to find. (BZ#1535917)

Previously, the secrets page in the web console did not display labels. You can now view the labels for a secret like other resources. (BZ#1545828)

Sometimes the web console displayed a process template page even if you did not have permissions to process templates. If you tried to process the template, an error displayed. Now you can no longer view process templates if you cannot process them. (BZ#1510786)

Previously, the Clear Changes button did not correctly clear edits to the Environment From variables in the web console environment variable editor. The button now correctly resets edits to Environment From variables. (BZ#1515527)

By default, dialogs in the web console can be dismissed by clicking in the negative space surrounding the dialog. IAs a result, the warning dialog could be inadvertently dismissed. With this bug fix, the warning dialog’s configuration was changed so that it can only be dismissed by clicking one of the buttons in the dialog. The warning dialog can no longer be inadvertently dismissed by the user, as clicking one of the dialog’s buttons is now required in order to close the dialog. (BZ#1525819)

Due to a fault in the scheduler implementation, the ScheduledImageImportMinimumIntervalSeconds setting was not correctly observed, causing OpenShift Container Platform to attempt to import scheduled images at the wrong intervals. With this bug fix, the issue is now resolved. (BZ#1515058)

The OpenShift Container Platform node was not waiting long enough for the VNID while the master assigns the VNID and it could take a while to propagate to the node. As a result, pod creation fails. Increase the timeout from 1 to 5 seconds for fetching VNID on the node. This bug fix allows pod creation to succeed. (BZ#1509799)

It is now possible to specify a subnet length as part of the EGRESS_SOURCE variable passed to an egress router (for example, 192.168.1.100/24 rather than 192.168.1.100). In some network configurations (such as if the gateway address was a virtual IP that might be backed by one of several physical IPs at different times), ARP traffic between the egress router and its gateway might not function correctly if the egress router is not able to send traffic to other hosts on its local subnet. By specifying EGRESS_SOURCE with a subnet length, the egress router setup script will configure the egress pod in a way that will work with these network setups. (BZ#1527602)

In some circumstances, iptables rules could become reordered in a way that would cause the per-project static IP address feature to stop working for some IP addresses. (For most users, egress IP addresses that ended with an even number would continue to work, but egress IP addresses ending with an odd number would fail.) Therefore, external traffic from pods in a project that was supposed to use a per-project static IP address would end up using the normal node IP address instead. The iptables rules are changed so that they now have the expected effect even when they get reordered. With this bug fix, the per-project static egress IP feature now works reliably. (BZ#1527642)

Previously, the egress IP initialization code was only run when doing a full SDN setup, and not when OpenShift services were restarted and found any existing running SDN. This resulted in failure to create new per-project static egress IPs (HostSubnet.EgressIPs). This issue is now fixed and per-project static egress IPs works correctly after a node restart. (BZ#1533153)

Previously, OpenShift was setting colliding host-subnet values, which resulted in pod IP network to became unavailable across the nodes. This was because the stale OVS rules were not cleared during node startup. This is now fixed and the stale OVS rules are cleared on node startup. (BZ#1539187)

With previous version, if an static IP addressed was removed from a project and then added back to the same project, it did not worked correctly. This is now fixed, removing and re-adding static egress IPs works. (BZ#1547899)

Previously, when OpenShift was deployed on OpenStack, there were few required iptables rules that were not created automatically, which resulted in errors in pop-to-pod communication between pods on different nodes. The Ansible OpenShift installer now sets the required iptables rules automatically. (BZ#1493955)

There was a race condition in the startup code that relied on the node setup, setting a field that the userspace proxy needed. When the network plugin was not used (or if it was fast) the userspace proxy setup ran sooner and resulted in reading a nil value for the IP address of the node. Later when the proxy (or the unidler which uses it) was enabled, it would crash because of the nil IP address value. This issue is now fixed. A retry loop is added that waits for the IP address value to be set and the userspace proxy and unidler work as expected. (BZ#1519991)

In some circumstances, nodes were receiving a duplicate out-of-order HostSubnet deleted event from the master. During processing of this duplicate event, the node ended up deleting OVS flows corresponding to an active node, disrupting communications between these two nodes. In the latest version. the HostSubnet event-processing now checks for and ignores duplicate events. Thus, the OVS flows are not deleted, and pods communicate normally. (BZ#1544903)

Previously, the openshift ex dockergc command to cleanup docker images, failed occasionally. This issue is now fixed. (BZ#1511852)

Previously, nested secrets did not get mounted in pod. This issue is now fixed. (BZ#1516569)

HAproxy versions earlier than version 1.9 dropped new connections during a reload. This issue is now fixed. By using HAproxy’s seamless reload feature, HAproxy now passes open sockets when reloading, fixing reload issues. fixed. (BZ#1464657)

There was a spurious error in system logs. The error Stat fs failed. Error: no such file or directory appeared in logs frequently. This was because of calling the syscall.Statfs function in code when the path does not exist. This issue is now fixed. (BZ#1511576)

Previously, a reject routes error message showed up when using router shards. This issue is now fixed and the rejected routes error messages are now suppressed in HAproxy if router shards are used. (BZ#1491717)

Previously, if creating a route with the host set to localhost, and if the ROUTER_USE_PROXY_PROTOCOL environment variable was not set to true, any route reloads would fail. This is because the hostname being set to the default resulted in mismatches in route configurations. The -H option is now available when using curl, meaning the health check does not pass the hostname when set to 'localhost', and routes reload successfully. (BZ#1542612)

Previously, updating TLS certificates was not possible for cluster administrators. Because it is an expected task of the cluster administrator, the role has been changed to update TLS certificates. (BZ#1524707)

Previously, the APBs for MariaDB, PostgreSQL, and MySQL were tagged as "databases" instead of "database". This is corrected with the tag "database" matching other services which is now properly shown in search results. (BZ#1510804)

Async bind and unbind is an experimental feature for the OpenShift Ansible broker (OAB) and is not supported or enabled by default. Red Hat’s officially released APBs (PostgreSQL, MariaDB, MySQL, and Mediawiki) also do not support async bind and unbind. (BZ#1548997)

Previously, the etcd server was not accessible when using the etcdctl command. This was caused by the tcp being set to “0.0.0.0” instead of the expected --advertise-client-urls value of the asb-etcd deployment configuration. The command had been updated and the etcd server is now accessible. (BZ#1514417)

Previously, the apb push -o command failed when using it outside the cluster. This was because the Docker registry service of the desired service was set to hit only the route used by internal operations. The appropriate Ansible playbook has been updated to point to the appropriate route instead. (BZ#1519193)

Previously, when typing asbd --help or asbd -h, the --help argument returned a code that was being misinterpreted as an error, resulting in errors printing out twice. The fix corrects errors to only print once and also to interpret the help command return code as valid. As a result, the help command now only prints once. (BZ#1525817)

Previously, setting the white-list variable in an RHCC registry would maintain searching for any options, even after those options are removed from the configuration. This was caused by an error in the white-list code. The error has been fixed by this bug. (BZ#1526887)

Previously, if the registry configuration did not have auth_type set to config error messages would appear. This bug ensures that registry configurations work correctly without the auth_type setting. (BZ#1526949)

Previously, the broker would return a 400 status code when the user did not have the permissions to execute a task instead of the 403 status code. This bug fixes the error, and the correct status code is now returned. (BZ#1510486)

Previously, any MariaDB configuration options were displayed with MySQL options. This is because MariaDB uses MySQL variables upstream. This bug fix ensures that, in terms of OpenShift, the variables are called out as MariaDB. (BZ#1510294)

Previously, OpenShift checked mounted NFS volume with root squash. OpenShift permissions while running as root were squashed to the 'nobody' user, who did not have permissions to access mounted NFS volume. This caused any OpenShift checks to fail, and it did not unmount NFS volumes. Now, OpenShift does not access mounted NFS volumes, and checks for mounts by parsing /proc filesystem. NFS volumes with root squash option are unmounted. (BZ#1518237)

Previously, when a node that had an OpenStack Cinder type of persistent volume attached was shut down or crashed, the attached volume did not detach.Consequence: Because the persistent volume was unavailable, the pods did not migrate from the failed node, and the volumes were inaccessible from other nodes and pods. Now a node fails, all of its attached volumes are detached after a time-out. (BZ#1523142)

Previously, downward API, secrets, ConfigMap, and projected volumes fully managed their content and did not allow any other volumes to be mounted on top of them. This meant that users could not mount any volume on top of the aforementioned volumes. With this bug fix, the volumes now touch only the files they create. As a result, users can mount any volume on top of the aforementioned volumes. (BZ#1430322)

The upgrade playbooks did not previously regenerate the registry certificate when upgrading from releases prior to 3.6, which lacked the name 'docker-registry.default.svc'. As such, the configuration variables were not updated to push to the registry via DNS. The 3.9 upgrade playbooks now regenerate the certificate when needed, ensuring that all environments upgraded to 3.9 now push to the registry via DNS. (BZ#1519060)

The etcd host validation now accepts one or more etcd hosts, allowing greater flexibility in the number of etcd hosts configured. The recommended number of etcd hosts is still 3. (BZ#1506177)

There is a known issue in the initial GA release of OpenShift Container Platform 3.9 that causes the installation and upgrade playbooks to consume more memory than previous releases. The node scale-up and installation Ansible playbooks may have consumed more memory on the control host (the system where you run the playbooks from) than expected due to the use of include_tasks in several places. This issue has been addressed with the release of RHBA-2018:0600; the majority of these instances have now been converted to import_tasks calls, which do not consume as much memory. After this change, memory consumption on the control host should be below 100MiB per host; for large environments (100+ hosts), a control host with at least 16GiB of memory is recommended. (BZ#1558672)