In Kubernetes, there are a few ways to resize statefulsets’s volumes. The most common way is to use the kube-statefulset-resize command. This command will resize all of the volumes in a statefulset, and then create new volumes if needed. If you want to resize only some of the volumes, you can use the –volume-count argument. This argument will tell Kubernetes how many volumes to resize at once. Here’s an example using the kube-statefulset-resize command: $ kubeconfig –statefulset=my-statefulset –name=my-container mynewvolumes $ kubeconfig –statefulset=my-statefulset –name=mynewcontainer myoldvolumes
Kubernetes StatefulSets are used to deploy stateful applications inside your cluster. Each Pod in the StatefulSet can access local persistent volumes that stick to it even after it’s rescheduled. This allows Pods to maintain individual state that’s separate from their neighbors in the set.
Unfortunately these volumes come with a big limitation: Kubernetes doesn’t provide a way to resize them from the StatefulSet object. The spec.resources.requests.storage property of the StatefulSet’s volumeClaimTemplates field is immutable, preventing you from applying any capacity increases you require. This article will show you how to workaround the problem.
Creating a StatefulSet
Copy this YAML and save it to ss.yaml:
Apply the YAML to your cluster with Kubectl:
You’ll need a storage class and provisioner in your cluster to run this example. It creates a StatefulSet that runs three replicas of an NGINX web server.
While this isn’t representative of when StatefulSets should be used, it’s adequate as a demo of the volume problems you can face. A volume claim with 1 Gi of storage is mounted to NGINX’s data directory. Your web content could outgrow this relatively small allowance as your service scales. However trying to modify the volumeClaimTemplates.spec.resources.requests.storage field to 10Gi will report the following error when you run kubectl apply:
This occurs because almost all the fields of a StatefulSet’s manifest are immutable after creation.
Manually Resizing StatefulSet Volumes
You can bypass the restriction by manually resizing the persistent volume claim (PVC). You’ll then need to recreate the StatefulSet to release and rebind the volume from your Pods. This will trigger the actual volume resize event.
First use Kubectl to find the PVCs associated with your StatefulSet:
There are three PVCs because there are three replicas in the StatefulSet. Each Pod gets its own individual volume.
Now use kubectl edit to adjust the capacity of each volume:
The PVC’s YAML manifest will appear in your editor. Find the spec.resources.requests.storage field and change it to your new desired capacity:
Save and close the file. Kubectl should report that the change has been applied to your cluster.
Now repeat these steps for the StatefulSet’s remaining PVCs. Listing your cluster’s persistent volumes should then show the new size against each one:
The claims will maintain the old size for now:
This is because the volume can’t be resized while Pods are still using it.
Recreating the StatefulSet
Complete the resize by releasing the volume claim from the StatefulSet that’s holding it. Delete the StatefulSet but use the orphan cascading mechanism so its Pods remain in your cluster. This will help minimize downtime.
Next edit your original YAML file to include the new volume size in the spec.resources.requests.storage file. Then use kubectl apply to recreate the StatefulSet in your cluster:
The new StatefulSet will assume ownership of the previously orphaned Pods because they’ll already meet its requirements. The volumes may get resized at this point but in most cases you’ll have to manually initiate a rollout that restarts your Pods:
The rollout proceeds sequentially, targeting one Pod at a time. This ensures your service remains accessible throughout.
Now your PVCs should show the new size:
Try connecting to one of your Pods to check the increased capacity is visible from within:
The Pod’s reporting the expected 10 Gi of storage.
Summary
Kubernetes StatefulSets let you run stateful applications in Kubernetes with persistent storage volumes that are scoped to individual Pods. However the flexibility this permits ends when you need to resize one of your volumes. This is a missing feature which currently requires several manual steps to be completed in sequence.
The Kubernetes maintainers are aware of the issue. There’s an open feature request to develop a solution which should eventually let you initiate volume resizes by editing a StatefulSet’s manifest. This will be much quicker and safer than the current situation.
One final caveat is that volume resizes are dependent on a storage driver that permits dynamic expansion. This feature only became generally available in Kubernetes v1.24 and not all drivers, Kubernetes distributions, and cloud platforms will support it. You can check whether yours does by running kubectl get sc and looking for true in the ALLOWVOLUMEXPANSION column of the storage driver you’re using with your StatefulSets.
