Authors: Matthew Doherty, Philipp Kuntz, Robert Gogolok
When extending the Kubernetes API with CustomResourceDefinitions you’ll come across the dilemma to clean up external resources when deleting a custom resource. Although you can create a custom resource simply to store and retrieve structured data, most of the time there is some entity involved, like custom controllers. The controller will manage this resource and create other external resources to handle the semantics of that resource. Those external resources should not live forever once the custom resource does not exist anymore.
In the following text, we’ll work with a custom resource example that represents a data service instance.
That data service instance could be, for example, an instance running a PostgreSQL database. That service instance might store data to an external blob store, for instance, AWS S3 during a backup operation. Once you want to get rid of this custom resource and therefore that service instance, you might want to clean up the backups that were created specifically for this data service instance (for example on AWS S3).
This is where Kubernetes Finalizers come into play and help to clean up external resources before the deletion of a custom resource. You can add finalizers to a custom resource that will prevent for example the kubectl tool from deleting it.
Let’s do a practical demo with Minikube on how finalizers can help to prevent the deletion of custom resources.
You can install Minikube using the Install Minikube instructions. Once it is up and running, you should be able to call kubectl get crd without an error.
In order to create a custom resource, we first need to create a custom resource definition. Please copy the following content to a file named customresourcedefinition.yaml:
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: serviceinstances.example.com spec: group: example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: service: type: string version: type: string scope: Namespaced names: plural: serviceinstances singular: serviceinstance kind: ServiceInstance shortNames: - si
It is a custom resource definition for the above-mentioned example of PostgreSQL service instances.
After creating the file, we’re ready to upload our custom resource definition to Kubernetes:
$ kubectl apply -f customresourcedefinition.yaml customresourcedefinition.apiextensions.k8s.io/serviceinstances.example.com created
Next we’ll create a custom resource fitting our custom resource definition. Create the following content in a file named customresource0.yaml:
apiVersion: "example.com/v1" kind: ServiceInstance metadata: name: my-new-service-instance0 finalizers: - my-finalizer.example.com spec: service: PostgreSQL version: "12"
Then we apply the custom resource using:
$ kubectl apply -f customresource0.yaml serviceinstance.example.com/my-new-service-instance0 created
Under metadata.finalizers we’ve added an entry for a finalizer called my-finalizer.example.com.
So far this doesn’t play a role and a new ServiceInstance resource has been created with the name my-custom-resource0.
We can get the resource’s yaml representation using:
$ kubectl get si my-new-service-instance0 -o yaml ... apiVersion: example.com/v1 kind: ServiceInstance metadata: ... creationTimestamp: "2020-09-09T21:36:56Z" finalizers: - my-finalizer.example.com ... name: my-new-service-instance0 ... spec: service: PostgreSQL version: "12"
Let’s try to delete the resource using:
$ kubectl delete -f customresource0.yaml serviceinstance.example.com "my-new-service-instance0" deleted
After outputting the delete line, kubectl is hanging.
In another shell we can now output the yaml representation of that custom resource again using:
$ kubectl get si my-new-service-instance0 -o yaml apiVersion: example.com/v1 kind: ServiceInstance metadata: ... deletionTimestamp: "2020-09-09T21:52:00Z" ...
Kubernetes has added the field metadata.deletionTimestamp to signal the intention is to delete that resource. The finalizer entry we’ve added is preventing Kubernetes from deleting that custom resource.
In order to get rid of the resource, we need to remove the finalizer entry and signal this way we’ve removed external resources locked by that finalizer name.
Let’s edit the file customresource0.yaml and remove the finalizer. The file should now look similar to the following content:
apiVersion: "example.com/v1" kind: ServiceInstance metadata: name: my-new-service-instance0 spec: service: PostgreSQL version: "12"
Let’s apply the changes:
$ kubectl apply -f customresource0.yaml serviceinstance.example.com/my-new-service-instance0 configured
When we switch back to the hanging kubectl command, we can see it succeeded. The custom resource has been removed since the list of finalizers is empty.
The implications for Kubernetes are that all finalizers have been executed and have done their job.
Specifying finalizers can prevent a custom resource from deletion. This gives the opportunity to clean up external resources associated with the custom resource.
In a future article, we’ll extend our knowledge to Kubernetes operators and how to protect custom resources with finalizers during reconciliation.