Volumes in Kubernetes are mounted filesystems or files that are managed by the kubelet. These volumes are defined at the pod level and each container can specify which volume to mount and the mount location.
There are several types of volumes that may be used on Catalyst Cloud’s Managed Kubernetes, some common types include:
hostpath (pay special attention to the security implications)
To read more about using these volume types, refer to the Kubernetes Volume documentation.
On this page, we will show examples of ephemeral volumes using emptyDir, and persistent volumes using persistentVolumeClaims, which are provisioned using OpenStack Cinder our block storage service.
In Kubernetes, ephemeral volumes are tied to pods and their life cycles. They are the most basic storage abstraction, and containers within the pod mount these volumes and access them as if they were a local filesystem.
This also provides a mechanism for containers within a pod to be able to share data by mounting the same volume.
Note
Once a pod terminates, all ephemeral volumes and associated data within it are gone.
To use a volume, a Pod specifies a list of volumes and where to mount them in
each container. The containers themselves see these presented as file systems.
Kubernetes supports several different volume types , for
this example we will use an emptyDir
volume which is essentially an empty
directory bind mounted from the host.
apiVersion: v1
kind: Pod
metadata:
name: test-pod
spec:
containers:
- image: test-image-web
name: test-web
volumeMounts:
- mountPath: /data
name: shared-volume
- image: test-image-logger
name: test-logger
volumeMounts:
- mountPath: /logs
name: shared-volume
volumes:
- name: shared-volume
emptyDir: {}
Here we can see we have two containers in the pod that mount the same volume, shared-volume. The test-web container mounts it on /data, while the test-logger container mounts it on /logs. Both containers can then read and write to the same filesystem.
PersistentVolumes
on the other hand, exist as separate resources within
Kubernetes.
They work differently to emphemeral volumes, in that pods need to claim a volume
based on a StorageClass
and will retain it throughout their lifetime
until it is released. PersistentVolumes
also have the option to
retain the volume even if the pods using it are destroyed.
The cluster administrator can create pre-configured static PersistentVolumes
(PV) that define a particular size and type of volume and these in turn can be
utilised by end users, such as application developers, via a
PersistentVolumeClaim
(PVC).
If a PersistentVolumeClaim
is made and no matching PersistentVolume
is found, Kubernetes can attempt to dynamically provision the storage based
on the volume claim. If no storage class is defined then the default storage
class will be used.
Catalyst Cloud provides pre-defined Storage Classes for the block storage tiers in each region. See Volume Tiers.
The storage class names and their availability by region are as follows:
Storage class |
nz-por-1 |
nz-hlz-1 |
nz_wlg_2 |
---|---|---|---|
b1.standard |
available |
available |
available |
b1.sr-r3-nvme-1000 |
available |
available |
|
b1.sr-r3-nvme-2500 |
available |
available |
|
b1.sr-r3-nvme-5000 |
available |
available |
Let’s look at the steps involved in dynamically allocating a PersistentVolume
to
a pod.
First, we create a definition file for a PersistentVolumeClaim
using a
storage class from the table above. We also need to specify a name for
the claim and a size for the volume. In our example we will use the following:
storage class : b1.sr-r3-nvme-1000
volume name : test-persistentvolumeclaim
volume size : 1 GiB
# pvc1.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-persistentvolumeclaim
spec:
accessModes:
- ReadWriteOnce
storageClassName: b1.sr-r3-nvme-1000
resources:
requests:
storage: 1Gi
Now create the claim for the volume in Kubernetes.
$ kubectl create -f pvc1.yaml
persistentvolumeclaim/test-persistentvolumeclaim created
We should soon be able to see this PVC transition to Bound status, as a
PersistentVolume
is created and bound, and a Cinder volume created in OpenStack.
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test-persistentvolumeclaim Bound pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84 1Gi RWO b1.sr-r3-nvme-1000 17s
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84 1Gi RWO Retain Bound default/test-persistentvolumeclaim b1.sr-r3-nvme-1000 17s
To access this from within a pod we need to add a volumes
entry
specifying the PersistentVolumeClaim
and give it a name. Then we add a
volumeMounts
entry to the container that links to the PVC by its name and
finally a mountPath
entry that defines the target path for the volume to
be mounted in the container.
# pvc1-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-pv-test
spec:
containers:
- name: test-storage-container
image: nginx:latest
ports:
- containerPort: 8080
name: "http-server"
volumeMounts:
- mountPath: "/data"
name: test-persistentvolume
volumes:
- name: test-persistentvolume
persistentVolumeClaim:
claimName: test-persistentvolumeclaim
Create the new pod.
$ kubectl create -f pvc1-pod.yaml
pod/pod-pv-test created
Once the pod is available, exec into it with a bash shell to confirm that the new volume is mounted on /data.
$ kubectl exec -it pod-pv-test -- /bin/bash
root@pod-pv-test:/# mount | grep data
/dev/vdd on /data type ext4 (rw,relatime,seclabel)
If we describe the pod we can see under the Mounts: section that it has a volume mounted from the storage class block-storage-class and the Volumes: section shows the related persistent volume claim for this storage.
$ kubectl describe pod pod-pv-test
Name: pod-pv-test
Namespace: default
Priority: 0
Service Account: default
Node: cluster1-afjuly77v4gr-node-0/10.0.0.14
Start Time: Wed, 31 Jan 2024 22:49:45 +0000
Labels: <none>
Annotations: cni.projectcalico.org/containerID: d0e0273a9c017d9beb9476d886bd028c953ef8e39e730a7f407eb411f39d7dca
cni.projectcalico.org/podIP: 10.100.130.208/32
cni.projectcalico.org/podIPs: 10.100.130.208/32
Status: Running
IP: 10.100.130.208
IPs:
IP: 10.100.130.208
Containers:
test-storage-container:
Container ID: containerd://5fcb09543ec5202f1a2ee56b2086aaabe8291fd07234e75a1bf10f40d198a106
Image: nginx:latest
Image ID: docker.io/library/nginx@sha256:4c0fdaa8b6341bfdeca5f18f7837462c80cff90527ee35ef185571e1c327beac
Port: 8080/TCP
Host Port: 0/TCP
State: Running
Started: Wed, 31 Jan 2024 22:50:08 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/data from test-persistentvolume (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2npp (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
test-persistentvolume:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: test-persistentvolumeclaim
ReadOnly: false
kube-api-access-p2npp:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m default-scheduler Successfully assigned default/pod-pv-test to cluster1-afjuly77v4gr-node-0
Normal SuccessfulAttachVolume 111s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84"
Normal Pulling 108s kubelet Pulling image "nginx:latest"
Normal Pulled 97s kubelet Successfully pulled image "nginx:latest" in 10.696765403s (10.69679197s including waiting)
Normal Created 97s kubelet Created container test-storage-container
Normal Started 97s kubelet Started container test-storage-container
When a PersistentVolume
is used as a resource within a cluster through the
creation of a PersistentVolumeClaim
it is important to know that the underlying
physical volume assigned to the claim will persist if the cluster is removed.
Note
The persistence of the underlying volume is not affected by the setting
of the StorageClass
Reclaim Policy in the scenario where the cluster is
deleted before removing resources.
If the PersistentVolumeClaim
resource was intentionally released prior to the
cluster being terminated however, the usual retention policy for that
storage class will apply.
The policy Retain is the default provided, and this allows for manual deletion of the underlying resource. This prevents accidental removal, and permits re-attachment to another Kubernetes cluster.
For some workloads, it will be preferable to use the Delete reclaim policy which
will remove both the automatically created PersistentVolume
and the underlying
Cinder volume when the PersistentVolumeClaim
is deleted.
Refer to the Kubernetes documentation on Persistent Volume Reclaiming for more details.
PersistentVolumes
created using the provided StorageClasses
can be resized
and Kubernetes since v1.11 supports this.
Note
PersistentVolumes
can only be expanded, not reduced in size.
On Catalyst Cloud you may need to apply the below patch to enable resizing, as ALLOWVOLUMEEXPANSION has not been enabled by default. The patch is idempotent and can be safely re-applied to all clusters.
$ kubectl patch storageclass b1.sr-r3-nvme-1000 b1.sr-r3-nvme-2500 b1.sr-r3-nvme-5000 b1.standard -p '{"allowVolumeExpansion": true}'
storageclass.storage.k8s.io/b1.sr-r3-nvme-1000 patched
storageclass.storage.k8s.io/b1.sr-r3-nvme-2500 patched
storageclass.storage.k8s.io/b1.sr-r3-nvme-5000 patched
storageclass.storage.k8s.io/b1.standard patched
Now listing StorageClasses
will show true in the ALLOWVOLUMEEXPANSION column.
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
b1.sr-r3-nvme-1000 (default) cinder.csi.openstack.org Retain Immediate true 6d3h
b1.sr-r3-nvme-2500 cinder.csi.openstack.org Retain Immediate true 6d3h
b1.sr-r3-nvme-5000 cinder.csi.openstack.org Retain Immediate true 6d3h
b1.standard cinder.csi.openstack.org Retain Immediate true 6d3h
To initiate a resize, edit the PersistentVolumeClaim
and set
spec.resources.requests.storage to an increased size.
$ kubectl edit pvc/test-persistentvolumeclaim
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
volume.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
creationTimestamp: "2024-01-31T22:42:26Z"
finalizers:
- kubernetes.io/pvc-protection
name: test-persistentvolumeclaim
namespace: default
resourceVersion: "1156755"
uid: bca8b8ef-01b2-408e-aba6-bf9bef249e84
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: b1.sr-r3-nvme-1000
volumeMode: Filesystem
volumeName: pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84
Once this has been done, there will be several failed events. This is because online resizing is not permitted and a pod has the PV mounted.
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
5s Warning ExternalExpanding persistentvolumeclaim/test-persistentvolumeclaim waiting for an external controller to expand this PVC
2s Normal Resizing persistentvolumeclaim/test-persistentvolumeclaim External resizer is resizing volume pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84
2s Warning VolumeResizeFailed persistentvolumeclaim/test-persistentvolumeclaim resize volume "pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84" by resizer "cinder.csi.openstack.org" failed: rpc error: code = Internal desc = Could not resize volume "127fd890-8bb5-4962-92f6-bf498a8881c5" to size 2: Expected HTTP response code [202] when accessing [POST https://api.catalystcloud.nz:8776/v3/b6170e3eab4d44428c879485de7bee98/volumes/127fd890-8bb5-4962-92f6-bf498a8881c5/action], but got 406 instead
To complete the resize, re-create the pod that has the PersistentVolume
mounted; either
by deleting it and re-creating or by scaling the associated deployment down and back up.
In this example, we do not have a Deployment
, so we will delete and re-create the pod.
$ kubectl delete -f pvc1-pod.yaml
pod "pod-pv-test" deleted
$ kubectl create -f pvc1-pod.yaml
pod/pod-pv-test created
When the PersistentVolume
is mounted on the new pod, it will be resized by
Cinder CSI, and the kubelet will resize the filesystem. Then the pod will be
started with the new size.
We can see these events
and verify the size is reflected inside the new pod:
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
2m57s Normal Killing pod/pod-pv-test Stopping container test-storage-container
2m28s Warning ExternalExpanding persistentvolumeclaim/test-persistentvolumeclaim waiting for an external controller to expand this PVC
2m28s Normal Resizing persistentvolumeclaim/test-persistentvolumeclaim External resizer is resizing volume pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84
2m26s Normal FileSystemResizeRequired persistentvolumeclaim/test-persistentvolumeclaim Require file system resize of volume on node
2m15s Normal Scheduled pod/pod-pv-test Successfully assigned default/pod-pv-test to cluster1-afjuly77v4gr-node-0
2m7s Normal SuccessfulAttachVolume pod/pod-pv-test AttachVolume.Attach succeeded for volume "pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84"
2m5s Normal FileSystemResizeSuccessful persistentvolumeclaim/test-persistentvolumeclaim MountVolume.NodeExpandVolume succeeded for volume "pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84" cluster1-afjuly77v4gr-node-0
2m5s Normal FileSystemResizeSuccessful pod/pod-pv-test MountVolume.NodeExpandVolume succeeded for volume "pvc-bca8b8ef-01b2-408e-aba6-bf9bef249e84" cluster1-afjuly77v4gr-node-0
2m4s Normal Pulling pod/pod-pv-test Pulling image "nginx:latest"
2m3s Normal Pulled pod/pod-pv-test Successfully pulled image "nginx:latest" in 1.579627165s (1.579648593s including waiting)
2m3s Normal Created pod/pod-pv-test Created container test-storage-container
2m2s Normal Started pod/pod-pv-test Started container test-storage-container
$ kubectl exec -it pod-pv-test -- /bin/bash
root@pod-pv-test:/# mount | grep data
/dev/vdd on /data type ext4 (rw,relatime,seclabel,mb_optimize_scan=0)
root@pod-pv-test:/# df -h /data
Filesystem Size Used Avail Use% Mounted on
/dev/vdd 3.0G 24K 3.0G 1% /data
Assume we had previously created a cluster that had a Pod with a
PersistentVolume
mounted on it. If the cluster was deleted without deleting
the PersistentVolume
, or if the PersistentVolume
was set to Retain, we can still
find the Cinder volume and use it within another Kubernetes cluster in the same
cloud project and region.
First we need to query our project to identify the volume in question and retrieve its UUID.
$ openstack volume list
+--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+
| 6b1903ea-d1aa-452d-93cc-xxxxxxxxxxxx | kubernetes-dynamic-pvc-1e3b558f-3945-11e9-9776-xxxxxxxxxxxx | available | 1 | |
+--------------------------------------+-------------------------------------------------------------+-----------+------+-------------+
Once we have the ID value for the volume in question we can create a new
PersistentVolume
resource in our cluster and link it specifically to that
volume.
# pv-existing.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: existing-pv
labels:
type: existing-pv
spec:
capacity:
storage: 1Gi
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
cinder:
fsType: ext4
volumeID: 119f5f68-585e-41dd-b1a4-xxxxxxxxxxxx
$ kubectl create -f pv-existing.yaml
persistentvolume/cinder-pv created
Once we have the PV created we need to create a corresponding
PersistentVolumeClaim
for that resource.
The key point to note here is that our claim needs to reference the specific
PersistentVolume
we created in the previous step. To do this we use a selector
with the matchLabels
argument to refer to a corresponding label that we had
in the PersistentVolume
declaration.
# pvc-existing-pv.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: existing-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
selector:
matchLabels:
type: existing-pv
$ kubectl create -f pvc-existing-pv.yaml
persistentvolumeclaim/existing-cinder-pv-claim created
Finally we can create a new Pod that uses our PersistentVolumeClaim
to mount
the required volume on this pod.
# pod-with-existing-pv.yaml
apiVersion: v1
kind: Pod
metadata:
name: pod-exisitng-pv-test
spec:
volumes:
- name: existing-pv-storage
persistentVolumeClaim:
claimName: existing-pv-claim
containers:
- name: test-exisitng-storage-container
image: nginx:latest
ports:
- containerPort: 8080
name: "http-server"
volumeMounts:
- mountPath: "/data"
name: existing-pv-storage
$ kubectl create -f pod-with-existing-pv.yaml
pod/pod-cinder created
If we describe the pod we can see that it has now successfully mounted our volume as /data within the container.
$ kubectl describe pod pod-cinder
Name: pod-cinder
Namespace: default
Node: k8s-dev-pvc-test-3-y4gcy3oygsil-minion-0/10.0.0.5
Start Time: Wed, 27 Feb 2019 10:26:25 +1300
Labels: <none>
Annotations: <none>
Status: Pending
IP:
Containers:
cinder-storage-container:
Container ID:
Image: lingxiankong/alpine-test
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/data from cinder-storage (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-kjqdd (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cinder-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: existing-cinder-pv-claim
ReadOnly: false
default-token-kjqdd:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-kjqdd
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 9s default-scheduler Successfully assigned default/pod-cinder to k8s-dev-pvc-test-3-y4gcy3oygsil-minion-0
Normal SuccessfulAttachVolume 3s attachdetach-controller AttachVolume.Attach succeeded for volume "cinder-pv"
If it is necessary to access the data on the PersistentVolume
device without
creating a new cluster, the volume in question will need to be attached to an
existing cloud instance and then mounted as a new volume within the filesystem.
For further information on mounting an existing volume, see Creating and using volumes.