Skip to content

Exercise 1b — Monitoring and Telemetry

In this exercise we will explore lightweight Kubernetes monitoring and telemetry using:

  • kubectl top
  • metrics-server
  • live traffic generated with iperf3

The goal is to observe how workload behaviour appears through cluster metrics in real time.

Rather than treating monitoring as something separate from workloads, we will generate traffic ourselves and watch the cluster respond.

Why Monitoring Matters

Once workloads become distributed across multiple nodes and containers, it becomes difficult to understand cluster behaviour from logs alone.

Monitoring systems help answer questions such as:

  • Which workloads are consuming CPU?
  • Which nodes are under pressure?
  • Are applications healthy?
  • Where are bottlenecks occurring?

In larger Kubernetes environments this telemetry is often collected using systems such as Prometheus, Grafana, Loki and OpenTelemetry.

For this tutorial we will use the lighter-weight Kubernetes metrics pipeline already available in the cluster.

Part 1 — Kubernetes Metrics

If metrics-server is installed (see Extra reading), Kubernetes can expose live resource usage information through the Kubernetes API, including:

  • CPU usage,
  • memory consumption,
  • and node utilisation.

Inspect node metrics:

kubectl top nodes

Inspect pod metrics:

kubectl top pods

At the moment the cluster may appear relatively idle. We will now generate network traffic and workload activity to make the telemetry more interesting.

Part 2 — Deploy an iperf3 Server and expose it

kubectl create namespace iperf-demo
kubectl apply -f $RES_HOME/iperf3.yaml

Inspect it:

kubectl get pods,svc

Part 3 — Launch a Client Pod

Create an interactive client pod:

kubectl run iperf3-client \
    --image=workshop-tools:arm64 \
    -it --rm \
    --restart=Never -- sh
/ #
And verify DNS resolution: 
/ # nslookup iperf3-service

Part 4 — Generate Network Traffic

In the client Run a bandwidth test:

/ # iperf3 -c iperf3-service

You should see output similar to:

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00 sec   1.10 GBytes   945 Mbits/sec

This traffic is pod-to-service, routed through Kubernetes networking, and potentially crossing worker nodes.

Observing Cluster Metrics

We can observe some of the metrics using kubectl top. In the client, run iperf3 -c iperf3-service -P 8 -t 20 to send multiple network connections between pods. Whilst it is running, open another terminal and watch node metrics update live with:

watch kubectl top pods -n iperf-demo

You should notice slight increases to the CPU activity, as well as changes in memory usage, and workload activity across nodes.

Even though this is only a small cluster, the same principles apply to much larger Kubernetes environments. Even with the production level monitoring, kube-state metrics can still be very useful.

Optional: Inspecting Pod Placement

View where workloads are running:

kubectl get pods -o wide

Check:

  • which node the server pod is running on,
  • where the client pod landed,
  • whether traffic may be crossing worker nodes.

Clean-up

You can end the interactive client with ctrl-d then 

kubectl delete deployment iperf-server
kubectl delete service iperf-service
kubectl config set-context --curent --namespace=default
to remove the deployment including client and server pods, the service, and restore the default namespace for kubectl.

Summary

In production Kubernetes environments, telemetry is often collected and visualised using Prometheus for metrics collection and Grafana for dashboards and visualisation.

These systems allow for historical monitoring, alerts, benchmarking and cluster-wide observability. However, they also introduce additional operational overhead and resource consumption, which is why we are using the lighter-weight metrics pipeline for this tutorial.

In this exercise you explored Kubernetes telemetry and workload behaviour, using kubectl top to monitor mock traffic generated by iperf3.

You also saw how Kubernetes exposes operational metrics that can later be integrated into larger observability platforms such as Prometheus and Grafana.