Kubernetes: Finalizers and Custom Controllers

Authors: Matthew Doherty, Philipp Kuntz, Robert Gogolok

In the latest blog post, we provided an introduction to Kubernetes Finalizers.
In the introduction, we covered the basic mechanics of how finalizers allow controllers to implement asynchronous pre-delete hooks. Simply put they inform the Kubernetes control plane that action needs to take place before Kubernetes garbage collection logic can be performed.

In this article, we take finalizers a step further by performing the resulting actions of a resource deletion with the help of a custom controller.

Table of Contents

• Preparation
• Demo
• Conclusion

Preparation

Once again we rely on minikube. Please ensure it’s installed and running.

We will use the same custom resource scenario from our previous example. In the previous article, we manually applied our custom resource definition providing a collection of API objects to the Kubernetes API.

Here, our custom controller provides this `CustomResourceDefinition` to the Kubernetes API when we run the controller against the configured Kubernetes cluster as described in the demo section.

For verbosity, we provide the `CustomResourceDefinition` below for reference.

1apiVersion: apiextensions.k8s.io/v1
2kind: CustomResourceDefinition
3metadata:
4  # name must match the spec fields below, and be in the form: <plural>.<group>
5  name: serviceinstances.example.com
6spec:
7  # group name to use for REST API: /apis/<group>/<version>
8  group: example.com
9  # list of versions supported by this CustomResourceDefinition
10  versions:
11    - name: v1alpha1
12      # Each version can be enabled/disabled by Served flag.
13      served: true
14      # One and only one version must be marked as the storage version.
15      storage: true
16      schema:
17        openAPIV3Schema:
18          type: object
19          properties:
20            spec:
21              type: object
22              properties:
23                service:
24                  type: string
25                version:
26                  type: string
27  # either Namespaced or Cluster
28  scope: Namespaced
29  names:
30    # plural name to be used in the URL: /apis/<group>/<version>/<plural>
31    plural: serviceinstances
32    # singular name to be used as an alias on the CLI and for display
33    singular: serviceinstance
34    # kind is normally the CamelCased singular type. Your resource manifests use this.
35    kind: ServiceInstance
36    # shortNames allow shorter string to match your resource on the CLI
37    shortNames:
38    - si

In this article, we will be using a custom controller to demonstrate how a controller can perform an action when notified about the pending deletion of a service instance.

To use, clone the repository

1git clone https://github.com/anynines/kubernetes-controller-finalizers-example.git
2cd kubernetes-controller-finalizers-example

and start the controller via:

1make install && make run ENABLE_WEBHOOKS=false

This will run the custom controller that is associated with our custom resource definition. The design pattern of a custom resource and a controller is known as a Kubernetes Operator.

Now we continue with the practical demo.

Demo

In the following steps, we will demonstrate a simple case where the reconcile a loop within the controller handles the insertion and deletion of a finalizer for our service instance resources.

We begin by creating a file named `customresource0.yaml` with the following custom resource content:

1apiVersion: "example.com/v1alpha1"
2kind: ServiceInstance
3metadata:
4  name: my-new-service-instance0
5spec:
6  service: PostgreSQL
7  version: "12"

Notice that the finalizer does not exist within the above resource as we have delegated responsibility to the controller which ensures that the resource includes a finalizer during the first iteration of its reconcile loop.

Then we need to apply it:

1$ kubectl apply -f customresource0.yaml
2serviceinstance.example.com/my-new-service-instance0 created

The running controller adds the finalizer to the custom resource under the `metadata.finalizers` field.

The finalizer is named `my-finalizer.example.com`. This can be seen after applying the resource with the following command: 

1kubectl get serviceinstance -o yaml

You should now see the following included in the custom resource indicating that the controller has taken responsibility for the resource by inserting the finalizer into the resource of the `serviceinstance`.

1…
2    finalizers:
3    - my-finalizer.example.com
4…

This is performed by the following code in our controller:

1...
2  myFinalizerName := "my-finalizer.example.com"
3  if serviceinstance.ObjectMeta.DeletionTimestamp.IsZero() {
4    // Inserting finalizer if missing from resource
5    if err := r.insertFinalizerIfMissing(ctx, log, serviceinstance, myFinalizerName); err != nil {
6      return ctrl.Result{}, err
7    }
8  } else {
9...
10
11func (r *ServiceInstanceReconciler) insertFinalizerIfMissing(ctx context.Context, log logr.Logger, instance *examplev1alpha1.ServiceInstance, finalizerName string) error {
12  if !containsString(instance.GetFinalizers(), finalizerName) {
13    log.Info("Inserting my finalizer")
14    instance.SetFinalizers(append(instance.GetFinalizers(), finalizerName))
15    if err := r.Update(context.Background(), instance); err != nil {
16      log.Error(err, "Failed to insert my finalizer")
17      return err
18    }
19    log.Info("Inserted my finalizer")
20  }
21  return nil
22}

The code checks whether the resource is not being deleted. If this is the case and the resource is missing our finalizer, we update the resource to include our finalizer. We then immediately return from the Reconcile method since we updated the resource causing the loop to be triggered again.

Now let’s try to delete the resource using:

1$ kubectl delete -f customresource0.yaml
2serviceinstance.example.com "my-new-service-instance0" deleted

This time you will notice that the resource can delete without kubectl hanging waiting for manual removal of the finalizer from the resource.

In the custom controller logs you should see the following log output:

2020-09-24T00:46:40.727+0200 INFO controllers.ServiceInstance Removing external resources {“serviceinstance”: “default/my-new-service-instance0”}
2020-09-24T00:46:40.727+0200 INFO controllers.ServiceInstance Removed external resources {“serviceinstance”: “default/my-new-service-instance0”}
2020-09-24T00:46:40.727+0200 INFO controllers.ServiceInstance Removing my finalizer {“serviceinstance”: “default/my-new-service-instance0”}
2020-09-24T00:46:40.737+0200 INFO controllers.ServiceInstance Removed my finalizer {“serviceinstance”: “default/my-new-service-instance0”}
2020-09-24T00:46:40.738+0200 DEBUG controller Successfully Reconciled {“reconcilerGroup”: “example.com, “reconcilerKind”: “ServiceInstance”, “controller”: “serviceinstance”, “name”: “my-new-service-instance0”, “namespace”: “default”}
2020-09-24T00:46:40.738+0200 INFO controllers.ServiceInstance serviceinstance resource not found. Ignoring since object must be deleted {“serviceinstance”: “default/my-new-service-instance0”}
2020-09-24T00:46:40.738+0200 DEBUG controller Successfully Reconciled {“reconcilerGroup”: “example.com”, “reconcilerKind”: “ServiceInstance”, “controller”: “serviceinstance”, “name”: “my-new-service-instance0”, “namespace”: “default”}

The reason for this is the custom controller came to our aid and provided clean-up the logic in the event of a deletion on the resource:

1myFinalizerName := "my-finalizer.example.com"
2  if serviceinstance.ObjectMeta.DeletionTimestamp.IsZero() {
3    ...
4  } else {
5    // The object is being deleted.
6    if err := r.removeFinalizerIfExists(ctx, log, serviceinstance, myFinalizerName); err != nil {
7      return ctrl.Result{}, err
8    }
9  }
10    ...
11
12// removeFinalizerIfExists removes finalizer and updates resource if specified
13// finalizer exists
14func (r *ServiceInstanceReconciler) removeFinalizerIfExists(ctx context.Context, log logr.Logger, instance *examplev1alpha1.ServiceInstance, finalizerName string) error {
15  if containsString(instance.GetFinalizers(), finalizerName) {
16    log.Info("Removing external resources")
17    if err := r.deleteExternalResources(instance); err != nil {
18      log.Error(err, "Failed to remove external resources")
19      return err
20    }
21    log.Info("Removed external resources")
22
23    log.Info("Removing my finalizer.")
24    instance.SetFinalizers(removeString(instance.GetFinalizers(), finalizerName))
25    if err := r.Update(ctx, instance); err != nil {
26      log.Error(err, "Failed to remove my finalizer")
27      return err
28    }
29    log.Info("Removed my finalizer")
30  }
31  return nil
32}

The code determines if the resource is being deleted and removes the finalizer idempotently from the resource if it exists. This ensures that any clean-up logic that the controller sees as necessary, such as removal of external resources are performed before the controller relinquishing responsibility for the resource.

Conclusion

In this case, the logic is very simple, but we could imagine more complex scenarios. If the deletion timestamp exists in the resource remove the finalizer so that Kubernetes garbage collection can take place. This serves as an illustration of the power that custom controllers can provide.

In the deployment of a more complex application or data service, we may have other operations in mind that can be automated using this approach.