Develop on Kubernetes Series — Demystifying the For vs Owns vs Watches controller-builders in controller-runtime

Hi everyone, it’s been a while. Last year was hectic and my blog writing reduced quite substantially. But since the past few days I was really missing the good ol’ days where I used to just learn something new and pen that down over a blog post to bolster my understanding as well as support other learners of the same topic as well.

And in the middle of getting nostalgic about those good ol’ days, I decided to write a blog post on a topic which nagged me in the past when I started with Kubernetes operator/controller-dev.

It’s about difference between For() , Owns() and Watches() controller builders whenever a controller is registered withSetupWithManager .

This blog post would explain the theoretical side of these controller builders as well as their practical with a real-life example and corresponding code-snippets.

https://raw.githubusercontent.com/cncf/artwork/master/projects/kubernetes/stacked/all-blue-color/kubernetes-stacked-all-blue-color.png

Heads up

Before proceeding, I’d like to give a heads up on one thing: this blog post would assume that you are already aware of Kubernetes, the controller pattern, writing controllers, etc.

This blog post is dedicated towards diving into a topic which is beneficial to harness whenever you write (kinda) complicated controllers watching and reacting to, not one but, multiple resource types or events in general.

Let’s continue

kubebuilder is a beautiful thing. It bootstraps all the boilerplate piece of code for us whenever we want to write an operator/controller.

If you pay attention to the magically auto-generated code by kubebuilder, you’d see that for the controller you are writing, there would be a method defined like this

This method is fairly readable. It hooks up your controller to the manager’s scheme and client. So that whenever the manager is started, the controllers hooked to it also get started and get into the action of reconciling across the cluster.

Let’s take a look at theFor keyword. If you had asked me when I read this method first time, I would have easily guessed that the MyControlleris being hooked to reconcile against the MyCustomResource{} type. So, any create/update/delete happening with a MyCustomResource object in the cluster would trigger the Reconcile() method of MyController .

That’s great!

But what if MyController is not thaatt simple. What if it’s one heck of a complicated controller which is creating and owning other resources like Pods and trying to watch and reconcile them as well.

This is where things become less “guessable” and more fun ;)

Let’s work with an example

Let’s consider our sweet MyController . This controller is special and a bit crazy. What it does is that whenever aMyCustomResource object is created in the cluster, it creates annginx:latest pod with the labels foo: bar . But this controller is very nitpicky. It doesn’t want any add/update/delete happening to the labels ofthis pod.

This means that our controller wants to watch these pods and fix them whenever their labels are changed from foo: bar .

Hmm, how can we do that? :/

One step at a time

First of all, let’s only take care about creating the pod whenever MyCustomResource object is created.

Our Reconcile Method would take the following steps to do so:

• Get the MyCustomResourceobject, exit if it IsNotFound. Else,
• Get the name and namespace of that object.
• Create a nginx:latest pod with the same name and namespace, and labels foo: bar

You might be noticing a ton of flaws here like missing finalizers resolution, missing status reporting, etc. I won’t be focusing on that stuff for this blog as the purpose of this blog is around just digging into the aspects of For() , Owns() and Watches() :)

The above Reconcile()method will be triggered whenever a create/update/delete would happen with a MyCustomResource .

But what if

Someone goes into the cluster and manually removes or updates the foo: bar label. Our beloved MyController wouldn’t want that and would rather expect to be instantly triggered again and undergo through a reconciliation to bring the desired state of the pod back with the desired labels foo: bar .

But does this mean that our MyController needs to watch every Pod in the cluster?

Nope, MyController only cares about the pods it creates and doesn’t care about any other ad-hoc pods.

So, how do we encode some information in the child pods conveying that they were created by MyController

OwnerReferences!

Just prior to creating the Pod, the MyController can attach an ownerReference to the Pod with the owner being the MyCustomResource which is the parent of that Pod.

We do this the following way:

With this code, any Pod which will be created from the Reconcile() method would look like this:

apiVersion: v1
kind: Pod
metadata:
    name: somename
    namespace: somenamespace
    labels:
      foo: bar
    ownerReferences:
    - apiVersion: yash.com/v1alpha1
      controller: true
      kind: MyCustomResource
      name: <metadata.name of the owner mycustomresource>
      uid: <metadata.uid of the owner mycustomeresource>
spec:
     containers:
     - image: nginx:latest
       name: nginx-container

But this ain’t enough

The above code is just setting the MyCustomResource as the owner of the Pod. What we want is that whenever one of such pods is fiddled with, the above reconciliation loop is retriggered against that pod’s owner MyCustomResource .

And we achieve that with Owns() under the SetupWithManager method.

So, the SetupWithManager method would look like this:

The Owns(&corev1.Pod{}) is telling the manager to trigger MyController ‘s Reconcile()not only when MyCustomResource is fiddled with but also when any Pod with an ownerReference to a MyCustomResource is created/updated/deleted.

FYI, the req attribute in the Reconcile() method would contain the name/namespace of the owner MyCustomResource because our controller/reconciler, at the programmatic level, is only getting triggered For() one resource type which, in our case, is the MyCustomResource type.

We forgot one more thing

We need the piece of code in our Reconcile() method to NOT blindly create the Pod but check for its existence and labels first and only then, create/update the Pod accordingly.

After adding the above logic, the final code for Reconcile method would look like this:

So, TL;DR

• For(<ownerType>) — Tells the manager to trigger the Reconcile(...) whenever any object of <ownerType> is created/updated/deleted
• Owns(<childType>) — Tells the manager to trigger the Reconcile(...) whenever any object of <childType> having an ownerReference to <ownerType> is created/updated/deleted.

That’s pretty much it

Or is it?

We forgot to talk about Watches()

Just like For and Owns , we have one more controller-builder called Watches which is used to tell the manager, how/when to trigger the controller’s Reconcile(...) method.

But why do we need Watches() if we already have For() and Owns() ?

That’s because Watches(...) is very powerful at the cost of being slightly intricate to understand/write. With it, you can configure any possible event happening in the cluster to trigger the controller’s Reconcile() method.

Heck! You can even trigger the controller’s Reconcile() method for some outside-the-cluster event happening like some webhook getting triggered or just the clock hitting 12:00 AM at the night.

The method signature for Watches() looks like this:

Watches(source.Source, handler.EventHandler, ...)

• source.Source — represents what will be the source of the event which will trigger the reconciliation loop. Say, if you wanna trigger the reconciliation loop anytime a ConfigMap is created, you’d define it as Watches(&source.Kind{Type: &corev1.ConfigMap{}}, ..., ...)
• handler.EventHandler — represents which request will be exactly triggered (enqueued) against the Reconcile(req ctrl.Request) method. controller-runtime already provides us with readymade eventHandlers like handler.EnqueueRequestForObject{} which enqueues a request containing the Name/Namespace of the source of the event. For example, Watches(&source.Kind{Type: &corev1.ConfigMap{}}, &handler.EnqueueRequestForObject{}, ...) would trigger a Request whenever a configMap is created/updated/deleted and that request would contain the name/namespace of that configmap triggering the request.

In fact, For() and Owns() , behind the scenes, are plainly implementations of Watches() only

For example, in our MyController ‘s example:

For(&corev1alpha1.MyCustomResource{}) == Watches(&source.Kind{Type: &corev1alpha1.MyCustomResource{}}, &handler.EnqueueRequestForObject{})

Owns(&corev1.Pod{}) == Watches(&source.Kind{Type: &corev1.Pod{}}, &handler.EnqueueRequestForOwner{IsController: true, OwnerType: &corev1alpha1.MyCustomResource{}})

Let’s play with a fun example

Say, I want to trigger a reconciliation loop for the MyCustomResource with the name foo in the default namespace every night at 12:00AM

We can easily do that with Watches() with source.Channel under the SetupWithManager method itself. The following code achieves that:

• reconciliationSourceChannel — This is the channel where we will send an event.GenericEvent containing the object which we want to be reconciled. We have written a goroutine which will periodically check the current time and only perform a “send” on this channel at 12:00AM.
• Watches(&source.Channel{Source: reconciliationSourceChannel, &handler.EnqueueRequestForObject{}}) — Tells the manager to trigger the Reconcile(...) method whenever a value is received at the reconciliationSourceChannel . This received value would be a event.GenericEvent containing the Object which is supposed to be reconciled which, in our case, is going to be the MyCustomResource object with the name foo and namespace default .

So, long story short
We send an event on the channelreconciliationSourceChannel at 12:00AM and the Watches(...) receives it and executes the Reconcile() method for the MyCustomResource with the name foo and namespace default

That’s all folks!

Thanks a lot friends for sticking till here. I hope this blog post was comprehensible and taught you something new today. Feel absolutely free to drop in any questions or comments or even reach out to me on any of my social media handles:

Twitter — https://twitter.com/yashkukreja98

Github — https://github.com/yashvardhan-kukreja

LinkedIn — https://www.linkedin.com/in/yashvardhan-kukreja (expect slow replies here as I mostly stay inactive on it :P)

Until next time,

Adios!