K8s DaemonSets

 

NOTES

  • Cluster : A working Kubernetes deployment.
  • Nodes :   virtual (compute machine) or a physical machine
  • Pods : A pod is the smallest and simplest unit in the Kubernetes and it is scheduled and orchestrated to run on nodes.
  • Daemon :traditionally refers to long-running background processes on a server that handles things like logging.

 

Kubernetes DaemonSets run a daemon container on each node in the cluster.

Kubernetes DaemonSets

  • are used to deploy system daemons such as log collectors and monitoring agents, which typically must run on every node.
  • ensure a copy of a Pod is running across a set of nodes in a Kubernetes cluster.

 

There are some similarities between DaemonSets and ReplicaSets and sometimes you would be confused when to use one over the other.

ReplicaSets should be used when your application is completely decoupled from the node and you can run multiple copies on a given node without definite thought e.g  we are more concerned about  maintaining a given number of replicas than exactly which node the Pods run on.

DaemonSets should be used when a single copy of your application must run on all or a subset of the nodes in the cluster.

 

 

Creating DaemonSets

In this post, we’ll set up Fluentd  logging agent as a DaemonSet on every node  in our minikube cluster.

Fluentd is a popular open-source data collector that is usually set up on  Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored.

Note: The kubectl  command-line and Minikube should be installed  on your local PC

Create a file called fluentd.yaml in your text editor and add the code below.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  labels:
    app: fluentd
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.12-1
        env:
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENTD_SYSTEMD_CONF
            value: disable
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

In the file above , we implemented the following

  • We configured Fluentd  environment variables
    • FLUENT_ELASTICSEARCH_PORT: We set this to the Elasticsearch port we configured earlier, 9200.
    • FLUENT_ELASTICSEARCH_SCHEME: We set this to http.
    • FLUENTD_SYSTEMD_CONF: We set this to disable to suppress output related to systemd .
  • Then we specifed a 512 MiB memory limit on the Fluentd Pod, and guarantee it 0.1vCPU and 200MiB of memory. Next, we mount the /var/log and /var/lib/docker/containers host paths into the container using the varlog  and varlibdockercontainers .
  • The final parameter we define in this block is terminationGracePeriodSeconds, which gives Fluentd 30 seconds to shut down gracefully upon receiving a SIGTERM signal.

Open your terminal and run the following command minikube start

oluchi/Desktop $ minikube start
😄  minikube v1.19.0 on Darwin 10.15.7
✨  Using the hyperkit driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing hyperkit VM for "minikube" ...
🎉  minikube 1.20.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.20.0
💡  To disable this notice, run: 'minikube config set WantUpdateNotification false'

🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
    ▪ Using image kubernetesui/dashboard:v2.1.0
    ▪ Using image kubernetesui/metrics-scraper:v1.0.4
🌟  Enabled addons: storage-provisioner, default-storageclass, dashboard
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl apply -f fluentd.yaml
daemonset.apps/fluentd created

Once the fluentd DaemonSet has been successfully created , you can query its current state using the kubectl describe command:

$ kubectl describe daemonset fluentd

Output:

Name:           fluentd
Selector:       app=fluentd
Node-Selector:  <none>
Labels:         app=fluentd
Annotations:    deprecated.daemonset.template.generation: 1
Desired Number of Nodes Scheduled: 1
Current Number of Nodes Scheduled: 1
Number of Nodes Scheduled with Up-to-date Pods: 1
Number of Nodes Scheduled with Available Pods: 1
Number of Nodes Misscheduled: 0
Pods Status:  1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app=fluentd
  Containers:
   fluentd:
    Image:      fluent/fluentd:v1.12-1
    Port:       <none>
    Host Port:  <none>
    Limits:
      memory:  512Mi
    Requests:
      cpu:     100m
      memory:  200Mi
    Environment:
      FLUENT_ELASTICSEARCH_PORT:    9200
      FLUENT_ELASTICSEARCH_SCHEME:  http
      FLUENTD_SYSTEMD_CONF:         disable
    Mounts:
      /var/lib/docker/containers from varlibdockercontainers (ro)
      /var/log from varlog (rw)
  Volumes:
   varlog:
    Type:          HostPath (bare host directory volume)
    Path:          /var/log
    HostPathType:  
   varlibdockercontainers:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:  
Events:
  Type    Reason            Age    From                  Message
  ----    ------            ----   ----                  -------
  Normal  SuccessfulCreate  3m17s  daemonset-controller  Created pod: fluentd-w6z2r

Delete the fluentd daemonset

$ kubectl delete -f fluentd.yaml
daemonset.apps "fluentd" deleted

 

We brushed the basics of K8S daemonset and gave a distinction between replicasets and deamonset, set a up a simple fluentd logging agent on a minikube cluster.

If you want to learn more about it, you can visit K8S docs

 

 

 

 

 

 

 

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *