1. Documentation

Dynamic Provisioning and Creating Storage Classes

Overview

The StorageClass resource object describes and classifies storage that can be requested, as well as provides a means for passing parameters for dynamically provisioned storage on demand. StorageClass objects can also serve as a management mechanism for controlling different levels of storage and access to the storage. Cluster Administrators (cluster-admin) or Storage Administrators (storage-admin) define and create the StorageClass objects that users can request without needing any intimate knowledge about the underlying storage volume sources.

The OpenShift Origin persistent volume framework enables this functionality and allows administrators to provision a cluster with persistent storage. The framework also gives users a way to request those resources without having any knowledge of the underlying infrastructure.

Many storage types are available for use as persistent volumes in OpenShift Origin. While all of them can be statically provisioned by an administrator, some types of storage are created dynamically using the built-in provider and plug-in APIs.

Available Dynamically Provisioned Plug-ins

OpenShift Origin provides the following provisioner plug-ins, which have generic implementations for dynamic provisioning that use the cluster’s configured provider’s API to create new storage resources:

Storage Type Provisioner Plug-in Name Required Configuration Notes

OpenStack Cinder

kubernetes.io/cinder

Configuring for OpenStack

AWS Elastic Block Store (EBS)

kubernetes.io/aws-ebs

Configuring for AWS

For dynamic provisioning when using multiple clusters in different zones, tag each node with Key=KubernetesCluster,Value=clusterid.

GCE Persistent Disk (gcePD)

kubernetes.io/gce-pd

Configuring for GCE

In multi-zone configurations, it is advisable to run one Openshift cluster per GCE project to avoid PVs from getting created in zones where no node from current cluster exists.

GlusterFS

kubernetes.io/glusterfs

Container Native Storage with GlusterFS

Container Native Storage (CNS) utilizes Heketi to manage Gluster Storage.

Ceph RBD

kubernetes.io/rbd

Configuring for OpenStack

Trident from NetApp

netapp.io/trident

Configuring for Trident

Storage orchestrator for NetApp ONTAP, SolidFire, and E-Series storage.

Any chosen provisioner plug-in also requires configuration for the relevant cloud, host, or third-party provider as per the relevant documentation.

Defining a StorageClass

StorageClass objects are currently a globally scoped object and need to be created by cluster-admin or storage-admin users. There are currently five plug-ins that are supported. The following sections describe the basic object definition for a StorageClass and specific examples for each of the supported plug-in types.

Basic StorageClass Object Definition

StorageClass Basic Object Definition
kind: StorageClass (1)
apiVersion: storage.k8s.io/v1beta1 (2)
metadata:
  name: foo (3)
  annotations: (4)
     ...
provisioner: kubernetes.io/plug-in-type (5)
parameters: (6)
  param1: value
  ...
  paramN: value
1 (required) The API object type.
2 (required) The current apiVersion.
3 (required) The name of the StorageClass.
4 (optional) Annotations for the StorageClass
5 (required) The type of provisioner associated with this storage class.
6 (optional) The parameters required for the specific provisioner, this will change from plug-in to plug-in.

StorageClass Annotations

To set a StorageClass as the cluster-wide default:

   storageclass.beta.kubernetes.io/is-default-class: "true"

This enables any Persistent Volume Claim (PVC) that does not specify a specific volume to automatically be provisioned through the default StorageClass

To set a StorageClass description:

   kubernetes.io/description: My StorgeClass Description

OpenStack Cinder Object Defintion

cinder-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: gold
provisioner: kubernetes.io/cinder
parameters:
  type: fast  (1)
  availability: nova  (2)
1 VolumeType created in Cinder. Default is empty.
2 Availability Zone. If not specified, volumes are generally round-robined across all active zones where the OpenShift Origin cluster has a node.

AWS ElasticBlockStore (EBS) Object Defintion

aws-ebs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/aws-ebs
parameters:
  type: io1 (1)
  zone: us-east-1d (2)
  iopsPerGB: "10" (3)
  encrypted: true (4)
  kmsKeyId: keyvalue  (5)
1 Select from io1, gp2, sc1, st1. The default is gp2. See AWS docs for valid ARN value.
2 AWS zone. If not specified, volumes are generally round-robined across all active zones where the OpenShift Origin cluster has a node.
3 Only for io1 volumes. I/O operations per second per GiB. The AWS volume plug-in multiplies this with the size of the requested volume to compute IOPS of the volume. The value cap is 20,000 IOPS, which is the maximum supported by AWS. See AWS documentation for further details.
4 Denotes whether to encrypt the EBS volume. Valid values are true or false.
5 (optional) The full Amazon Resource Name (ARN) of the key to use when encrypting the volume. If none is supplied but encrypted is true, AWS generates a key. See AWS docs for valid ARN value.

GCE PersistentDisk (gcePD) Object Defintion

gce-pd-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard  (1)
  zone: us-central1-a   (2)
1 Select either pd-standard or pd-ssd. The default is pd-ssd.
2 GCE zone. If not specified, volumes are generally round-robined across all active zones where the OpenShift Origin cluster has a node.

GlusterFS Object Defintion

glusterfs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: slow
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://127.0.0.1:8081" (1)
  restuser: "admin" (2)
  secretName: "heketi-secret" (3)
  secretNamespace: "default" (4)
  gidMin: "40000" (5)
  gidMax: "50000" (6)
1 Gluster REST service/Heketi service URL that provisions Gluster volumes on demand. The general format should be {http/https}://{IPaddress}:{Port}. This is a mandatory parameter for the GlusterFS dynamic provisioner. If the Heketi service is exposed as a routable service in the OpenShift Origin, it will have a resolvable fully qualified domain name and Heketi service URL. For additional information and configuration, See Container-Native Storage for OpenShift Container Platform.
2 Gluster REST service/Heketi user who has access to create volumes in the Gluster Trusted Pool.
3 Identification of a Secret instance that contains a user password to use when talking to the Gluster REST service. Optional; an empty password will be used when both secretNamespace and secretName are omitted. The provided secret must be of type "kubernetes.io/glusterfs".
4 The namespace of mentioned secretName. Optional; an empty password will be used when both secretNamespace and secretName are omitted. The provided secret must be of type "kubernetes.io/glusterfs".
5 Optional. The minimum value of GID range for the storage class.
6 Optional. The maximum value of GID range for the storage class.

When the gidMin and gidMax values are not specified, the volume is provisioned with a value between 2000 and 2147483647, which are defaults for gidMin and gidMax respectively. If specified, a unique value (GID) in this range (gidMin-gidMax) is used for dynamically provisioned volumes. The GID of the provisioned volume will be set to this value. It is required to run Heketi version 3 or later to make use of this feature. This GID is released from the pool when the subjected volume is deleted. The GID pool is per storage class, if 2 or more storage classes have GID ranges that overlap there will be duplicate GIDs dispatched by the provisioner.

When the persistent volumes are dynamically provisioned, the Gluster plug-in automatically creates an endpoint and a headless service of the name gluster-dynamic-<claimname>. When the persistent volume claim is deleted, this dynamic endpoint and service is deleted automatically.

Example of a Secret
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
  key: bXlwYXNzd29yZA==
type: kubernetes.io/glusterfs

Ceph RBD Object Definition

ceph-storageclass.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: fast
provisioner: kubernetes.io/rbd
parameters:
  monitors: 10.16.153.105:6789  (1)
  adminId: admin  (2)
  adminSecretName: ceph-secret  (3)
  adminSecretNamespace: kube-system  (4)
  pool: kube  (5)
  userId: kube  (6)
  userSecretName: ceph-secret-user   (7)
1 Ceph monitors, comma delimited. It is required.
2 Ceph client ID that is capable of creating images in the pool. Default is "admin".
3 Secret Name for adminId. It is required. The provided secret must have type "kubernetes.io/rbd".
4 The namespace for adminSecret. Default is "default".
5 Ceph RBD pool. Default is "rbd".
6 Ceph client ID that is used to map the Ceph RBD image. Default is the same as adminId.
7 The name of Ceph Secret for userId to map Ceph RBD image. It must exist in the same namespace as PVCs. It is required.

Trident Object Definition

trident.yaml
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: gold
provisioner: netapp.io/trident (1)
parameters: (2)
  media: "ssd"
  provisioningType: "thin"
  snapshots: "true"

Trident uses the parameters as selection criteria for the different pools of storage that are registered with it. Trident itself is configured separately.

1 For more information about installing Trident with OpenShift Origin, see the Trident documentation.
2 For more information about supported parameters, see the storage attributes section of the Trident documentation.

Additional Information and Examples