- Cluster : A working Kubernetes deployment.
- Nodes : virtual (compute machine) or a physical machine
- Pods : A pod is the smallest and simplest unit in the Kubernetes and it is scheduled and orchestrated to run on nodes.
- Daemon :traditionally refers to long-running background processes on a server that handles things like logging.
Kubernetes DaemonSets run a daemon container on each node in the cluster.
- are used to deploy system daemons such as log collectors and monitoring agents, which typically must run on every node.
- ensure a copy of a Pod is running across a set of nodes in a Kubernetes cluster.
There are some similarities between DaemonSets and ReplicaSets and sometimes you would be confused when to use one over the other.
ReplicaSets should be used when your application is completely decoupled from the node and you can run multiple copies on a given node without definite thought e.g we are more concerned about maintaining a given number of replicas than exactly which node the Pods run on.
DaemonSets should be used when a single copy of your application must run on all or a subset of the nodes in the cluster.
In this post, we’ll set up Fluentd logging agent as a DaemonSet on every node in our minikube cluster.
Fluentd is a popular open-source data collector that is usually set up on Kubernetes nodes to tail container log files, filter and transform the log data, and deliver it to the Elasticsearch cluster, where it will be indexed and stored.
kubectl command-line and
Minikube should be installed on your local PC
Create a file called
fluentd.yaml in your text editor and add the code below.
apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd labels: app: fluentd spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: containers: - name: fluentd image: fluent/fluentd:v1.12-1 env: - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENTD_SYSTEMD_CONF value: disable resources: limits: memory: 512Mi requests: cpu: 100m memory: 200Mi volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true terminationGracePeriodSeconds: 30 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
In the file above , we implemented the following
- We configured Fluentd environment variables
FLUENT_ELASTICSEARCH_PORT: We set this to the Elasticsearch port we configured earlier,
FLUENT_ELASTICSEARCH_SCHEME: We set this to
FLUENTD_SYSTEMD_CONF: We set this to
disableto suppress output related to
- Then we specifed a 512 MiB memory limit on the Fluentd Pod, and guarantee it 0.1vCPU and 200MiB of memory. Next, we mount the
/var/lib/docker/containershost paths into the container using the
- The final parameter we define in this block is
terminationGracePeriodSeconds, which gives Fluentd 30 seconds to shut down gracefully upon receiving a
Open your terminal and run the following command
oluchi/Desktop $ minikube start 😄 minikube v1.19.0 on Darwin 10.15.7 ✨ Using the hyperkit driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing hyperkit VM for "minikube" ... 🎉 minikube 1.20.0 is available! Download it: https://github.com/kubernetes/minikube/releases/tag/v1.20.0 💡 To disable this notice, run: 'minikube config set WantUpdateNotification false' 🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 ▪ Using image kubernetesui/dashboard:v2.1.0 ▪ Using image kubernetesui/metrics-scraper:v1.0.4 🌟 Enabled addons: storage-provisioner, default-storageclass, dashboard 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
$ kubectl apply -f fluentd.yaml daemonset.apps/fluentd created
Once the fluentd DaemonSet has been successfully created , you can query its current state using the kubectl describe command:
$ kubectl describe daemonset fluentd
Name: fluentd Selector: app=fluentd Node-Selector: <none> Labels: app=fluentd Annotations: deprecated.daemonset.template.generation: 1 Desired Number of Nodes Scheduled: 1 Current Number of Nodes Scheduled: 1 Number of Nodes Scheduled with Up-to-date Pods: 1 Number of Nodes Scheduled with Available Pods: 1 Number of Nodes Misscheduled: 0 Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed Pod Template: Labels: app=fluentd Containers: fluentd: Image: fluent/fluentd:v1.12-1 Port: <none> Host Port: <none> Limits: memory: 512Mi Requests: cpu: 100m memory: 200Mi Environment: FLUENT_ELASTICSEARCH_PORT: 9200 FLUENT_ELASTICSEARCH_SCHEME: http FLUENTD_SYSTEMD_CONF: disable Mounts: /var/lib/docker/containers from varlibdockercontainers (ro) /var/log from varlog (rw) Volumes: varlog: Type: HostPath (bare host directory volume) Path: /var/log HostPathType: varlibdockercontainers: Type: HostPath (bare host directory volume) Path: /var/lib/docker/containers HostPathType: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 3m17s daemonset-controller Created pod: fluentd-w6z2r
Delete the fluentd daemonset
$ kubectl delete -f fluentd.yaml daemonset.apps "fluentd" deleted
We brushed the basics of K8S daemonset and gave a distinction between replicasets and deamonset, set a up a simple fluentd logging agent on a minikube cluster.
If you want to learn more about it, you can visit K8S docs