Published at 22.12.2020
Authors: Matthew Doherty, Philipp Kuntz, Robert Gogolok
When extending the Kubernetes API with CustomResourceDefinitions you’ll come across the dilemma to clean up external resources when deleting a custom resource. Although you can create a custom resource simply to store and retrieve structured data, most of the time there is some entity involved, like custom controllers. The controller will manage this resource and create other external resources to handle the semantics of that resource. Those external resources should not live forever once the custom resource does not exist anymore.
In the following text, we’ll work with a custom resource example that represents a data service instance.
That data service instance could be, for example, an instance running a PostgreSQL database. That service instance might store data to an external blob store, for instance, AWS S3 during a backup operation. Once you want to get rid of this custom resource and therefore that service instance, you might want to clean up the backups that were created specifically for this data service instance (for example on AWS S3).
This is where Kubernetes Finalizers come into play and help to clean up external resources before the deletion of a custom resource. You can add finalizers to a custom resource that will prevent for example the kubectl tool from deleting it.
Table of Contents
Let’s do a practical demo with Minikube on how finalizers can help to prevent the deletion of custom resources.
You can install Minikube using the Install Minikube instructions. Once it is up and running, you should be able to call kubectl get crd without an error.
In order to create a custom resource, we first need to create a custom resource definition. Please copy the following content to a file named customresourcedefinition.yaml:
1apiVersion: apiextensions.k8s.io/v1 2kind: CustomResourceDefinition 3metadata: 4 name: serviceinstances.example.com 5spec: 6 group: example.com 7 versions: 8 - name: v1 9 served: true 10 storage: true 11 schema: 12 openAPIV3Schema: 13 type: object 14 properties: 15 spec: 16 type: object 17 properties: 18 service: 19 type: string 20 version: 21 type: string 22 scope: Namespaced 23 names: 24 plural: serviceinstances 25 singular: serviceinstance 26 kind: ServiceInstance 27 shortNames: 28 - si
It is a custom resource definition for the above-mentioned example of PostgreSQL service instances.
After creating the file, we’re ready to upload our custom resource definition to Kubernetes:
1$ kubectl apply -f customresourcedefinition.yaml 2customresourcedefinition.apiextensions.k8s.io/serviceinstances.example.com created
Next we’ll create a custom resource fitting our custom resource definition. Create the following content in a file named customresource0.yaml:
1apiVersion: "example.com/v1" 2kind: ServiceInstance 3metadata: 4 name: my-new-service-instance0 5 finalizers: 6 - my-finalizer.example.com 7spec: 8 service: PostgreSQL 9 version: "12"
Then we apply the custom resource using:
1$ kubectl apply -f customresource0.yaml 2serviceinstance.example.com/my-new-service-instance0 created
Under metadata.finalizers we’ve added an entry for a finalizer called my-finalizer.example.com.
So far this doesn’t play a role and a new ServiceInstance resource has been created with the name my-custom-resource0.
We can get the resource’s yaml representation using:
1$ kubectl get si my-new-service-instance0 -o yaml 2... 3apiVersion: example.com/v1 4kind: ServiceInstance 5metadata: 6 ... 7 creationTimestamp: "2020-09-09T21:36:56Z" 8 finalizers: 9 - my-finalizer.example.com 10 ... 11 name: my-new-service-instance0 12 ... 13spec: 14 service: PostgreSQL 15 version: "12"
Let’s try to delete the resource using:
1$ kubectl delete -f customresource0.yaml 2serviceinstance.example.com "my-new-service-instance0" deleted
After outputting the delete line, kubectl is hanging.
In another shell we can now output the yaml representation of that custom resource again using:
1$ kubectl get si my-new-service-instance0 -o yaml 2apiVersion: example.com/v1 3kind: ServiceInstance 4metadata: 5 ... 6 deletionTimestamp: "2020-09-09T21:52:00Z" 7 ...
Kubernetes has added the field metadata.deletionTimestamp to signal the intention is to delete that resource. The finalizer entry we’ve added is preventing Kubernetes from deleting that custom resource.
In order to get rid of the resource, we need to remove the finalizer entry and signal this way we’ve removed external resources locked by that finalizer name.
Let’s edit the file customresource0.yaml and remove the finalizer. The file should now look similar to the following content:
1apiVersion: "example.com/v1" 2kind: ServiceInstance 3metadata: 4 name: my-new-service-instance0 5spec: 6 service: PostgreSQL 7 version: "12"
Let’s apply the changes:
1$ kubectl apply -f customresource0.yaml 2serviceinstance.example.com/my-new-service-instance0 configured
When we switch back to the hanging kubectl command, we can see it succeeded. The custom resource has been removed since the list of finalizers is empty.
The implications for Kubernetes are that all finalizers have been executed and have done their job.
Specifying finalizers can prevent a custom resource from deletion. This gives the opportunity to clean up external resources associated with the custom resource.
In a future article, we’ll extend our knowledge to Kubernetes operators and how to protect custom resources with finalizers during reconciliation.
Check out the full series here: Kubernetes Finalizers