Kubernetes: Finalizers in Custom Resources

Authors: Matthew Doherty, Philipp Kuntz, Robert Gogolok

When extending the Kubernetes API with CustomResourceDefinitions you’ll come across the dilemma to clean up external resources when deleting a custom resource. Although you can create a custom resource simply to store and retrieve structured data, most of the time there is some entity involved, like custom controllers. The controller will manage this resource and create other external resources to handle the semantics of that resource. Those external resources should not live forever once the custom resource does not exist anymore. 

In the following text, we’ll work with a custom resource example that represents a data service instance.

That data service instance could be, for example, an instance running a PostgreSQL database. That service instance might store data to an external blob store, for instance, AWS S3 during a backup operation. Once you want to get rid of this custom resource and therefore that service instance, you might want to clean up the backups that were created specifically for this data service instance (for example on AWS S3).

This is where Kubernetes Finalizers come into play and help to clean up external resources before the deletion of a custom resource. You can add finalizers to a custom resource that will prevent for example the kubectl tool from deleting it.

Table of Contents

• Demo
• Conclusion

Demo

Let’s do a practical demo with Minikube on how finalizers can help to prevent the deletion of custom resources.

You can install Minikube using the Install Minikube instructions. Once it is up and running, you should be able to call kubectl get crd without an error.

In order to create a custom resource, we first need to create a custom resource definition. Please copy the following content to a file named customresourcedefinition.yaml:

1apiVersion: apiextensions.k8s.io/v1
2kind: CustomResourceDefinition
3metadata:
4  name: serviceinstances.example.com
5spec:
6  group: example.com
7  versions:
8    - name: v1
9      served: true
10      storage: true
11      schema:
12        openAPIV3Schema:
13          type: object
14          properties:
15            spec:
16              type: object
17              properties:
18                service:
19                  type: string
20                version:
21                  type: string
22  scope: Namespaced
23  names:
24    plural: serviceinstances
25    singular: serviceinstance
26    kind: ServiceInstance
27    shortNames:
28    - si

It is a custom resource definition for the above-mentioned example of PostgreSQL service instances. 

After creating the file, we’re ready to upload our custom resource definition to Kubernetes:

1$ kubectl apply -f customresourcedefinition.yaml
2customresourcedefinition.apiextensions.k8s.io/serviceinstances.example.com created

Next we’ll create a custom resource fitting our custom resource definition.  Create the following content in a file named customresource0.yaml:

1apiVersion: "example.com/v1"
2kind: ServiceInstance
3metadata:
4  name: my-new-service-instance0
5  finalizers:
6  - my-finalizer.example.com
7spec:
8  service: PostgreSQL
9  version: "12"

Then we apply the custom resource using:

1$ kubectl apply -f customresource0.yaml
2serviceinstance.example.com/my-new-service-instance0 created

Under metadata.finalizers we’ve added an entry for a finalizer called my-finalizer.example.com.

So far this doesn’t play a role and a new ServiceInstance resource has been created with the name my-custom-resource0.

We can get the resource’s yaml representation using:

1$ kubectl get si my-new-service-instance0 -o yaml
2...
3apiVersion: example.com/v1
4kind: ServiceInstance
5metadata:
6  ...
7  creationTimestamp: "2020-09-09T21:36:56Z"
8  finalizers:
9  - my-finalizer.example.com
10  ...
11  name: my-new-service-instance0
12  ...
13spec:
14  service: PostgreSQL
15  version: "12"

Let’s try to delete the resource using:

1$ kubectl delete -f customresource0.yaml
2serviceinstance.example.com "my-new-service-instance0" deleted

After outputting the delete line, kubectl is hanging.

In another shell we can now output the yaml representation of that custom resource again using:

1$ kubectl get si my-new-service-instance0 -o yaml
2apiVersion: example.com/v1
3kind: ServiceInstance
4metadata:
5  ...
6  deletionTimestamp: "2020-09-09T21:52:00Z"
7  ...

Kubernetes has added the field metadata.deletionTimestamp to signal the intention is to delete that resource. The finalizer entry we’ve added is preventing Kubernetes from deleting that custom resource.

In order to get rid of the resource, we need to remove the finalizer entry and signal this way we’ve removed external resources locked by that finalizer name.

Let’s edit the file customresource0.yaml and remove the finalizer. The file should now look similar to the following content:

1apiVersion: "example.com/v1"
2kind: ServiceInstance
3metadata:
4  name: my-new-service-instance0
5spec:
6  service: PostgreSQL
7  version: "12"

Let’s apply the changes:

1$ kubectl apply -f customresource0.yaml
2serviceinstance.example.com/my-new-service-instance0 configured

When we switch back to the hanging kubectl command, we can see it succeeded.  The custom resource has been removed since the list of finalizers is empty.

The implications for Kubernetes are that all finalizers have been executed and have done their job.

Conclusion

Specifying finalizers can prevent a custom resource from deletion. This gives the opportunity to clean up external resources associated with the custom resource.

In a future article, we’ll extend our knowledge to Kubernetes operators and how to protect custom resources with finalizers during reconciliation.