Monitor Complex Kubernetes Deployments: Harbor Example

This guide shows you how to set up monitoring for a complex Kubernetes deployment, using Harbor (an open-source container registry) as a practical example. You’ll learn what metrics to track, which tools to use, and how to configure dashboards and alerts so you can spot issues before they become outages.

What You’ll Need

A running Kubernetes cluster (v1.19 or later recommended)
Harbor deployed on that cluster (via Helm or operator)
kubectl configured to access your cluster
Helm v3 installed (for installing monitoring stack)
Basic familiarity with Prometheus, Grafana, and Kubernetes concepts

Step-by-Step

Install the Prometheus Stack
Use the kube-prometheus-stack Helm chart to deploy Prometheus, Alertmanager, and Grafana. This gives you a solid foundation for monitoring both the cluster and applications.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
Enable Harbor Metrics
Harbor exposes metrics at /metrics on its core service. If you installed Harbor via Helm, ensure metrics.enabled=true in your values file. For an existing deployment, edit the deployment to add the necessary annotations.
kubectl annotate deployment harbor-core prometheus.io/scrape="true" --overwrite
kubectl annotate deployment harbor-core prometheus.io/port="8080" --overwrite
Create a ServiceMonitor for Harbor
ServiceMonitor is a custom resource that tells Prometheus which services to scrape. Create a file harbor-servicemonitor.yaml with the following content, adjusting selectors to match your Harbor labels.
```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: harbor-monitor
  namespace: harbor
spec:
  selector:
    matchLabels:
      app: harbor
  endpoints:
  - port: http
    interval: 30s
    path: /metrics
```
Apply it with kubectl apply -f harbor-servicemonitor.yaml.
Verify Prometheus is Scraping
Port-forward to Prometheus and check the targets page.
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090
Open http://localhost:9090/targets and look for the Harbor target. It should be UP.
Set Up Grafana Dashboards
Import a pre-built Harbor dashboard (search Grafana.com for “Harbor”) or create your own. At minimum, track:
- Request rate and latency for API endpoints
- Storage usage per project
- Number of artifacts and repositories
- GC job duration and success rate
- Replication status and failures
To import, log into Grafana (default admin/admin), go to Dashboards → Import, and paste the dashboard ID.
Create Alerts for Critical Metrics
Define Prometheus alerting rules for common issues. For example, create a rule that fires when Harbor core is down for more than 5 minutes, or when storage usage exceeds 85%. Add these to the PrometheusRule custom resource.
```
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: harbor-alerts
  namespace: monitoring
spec:
  groups:
  - name: harbor.rules
    rules:
    - alert: HarborCoreDown
      expr: up{job="harbor-core"} == 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Harbor Core is down"
```
Apply it and configure Alertmanager to send notifications via email, Slack, or PagerDuty.
Monitor Kubernetes Components
Use the kube-prometheus-stack’s built-in dashboards for node, pod, and cluster metrics. Pay special attention to:
- CPU and memory usage of Harbor pods
- Disk I/O on nodes hosting registry storage
- Network traffic and errors
Set Up Log Aggregation (Optional but Recommended)
Deploy Loki or Elastic Stack to collect logs from Harbor and Kubernetes. This helps you correlate metrics with actual error messages. For a quick start, install Loki with Helm and add it as a data source in Grafana.

Common Mistakes

Forgetting to scrape Harbor’s metrics endpoint – Harbor doesn’t expose metrics by default in older versions; you must enable it.
Using wrong selectors in ServiceMonitor – If the labels don’t match, Prometheus won’t find the service. Check with kubectl get svc -n harbor --show-labels.
Scraping too frequently – Setting scrape interval to 1s can overload Harbor. Stick to 15-30s.
Ignoring persistent volume metrics – Harbor’s registry storage can fill up silently. Monitor PV usage.
Not setting up alerts for disk space – This is the most common cause of Harbor outages.

Tips

Use Helm values to enable metrics – When installing Harbor, set metrics.enabled=true and metrics.serviceMonitor.enabled=true if you’re using the Prometheus operator.
Start with a few key metrics – Don’t try to monitor everything at once. Focus on the ones that indicate health: request latency, error rate, storage.
Set up alerting on error rate – A spike in 5xx responses from Harbor’s API is a strong signal.
Use Grafana annotations – Mark deployments or config changes on dashboards to correlate with metric changes.
Test your alerts – Kill a Harbor pod and confirm the alert fires. Don’t wait for a real incident.

Troubleshooting

Prometheus shows no targets for Harbor – Check ServiceMonitor selector labels, namespace, and that the service exposes a metrics port. Try kubectl get servicemonitor -n harbor and inspect with kubectl describe.
Metrics endpoint returns 404 – Harbor’s metrics are on the core service, not the registry. Ensure you’re hitting the right path (/metrics) and port (usually 8080).
Grafana shows no data – Verify the data source is configured correctly and that Prometheus is scraping. Check the target’s UP status.
Alerts not firing – Check PrometheusRule syntax and that the rule is in the correct namespace. Use kubectl get prometheusrule -n monitoring and look at the Prometheus UI under Rules.
High CPU usage from monitoring – Reduce scrape interval or limit the metrics you collect. Use relabeling to drop unnecessary series.

Frequently Asked Questions

Does Harbor expose metrics by default?

No, you must enable the metrics endpoint during installation. In Helm, set metrics.enabled=true. For existing deployments, you may need to add a sidecar or modify the core service.

What is the difference between ServiceMonitor and PodMonitor?

ServiceMonitor scrapes a Kubernetes service, while PodMonitor scrapes individual pods. For Harbor, ServiceMonitor is usually easier because the service is stable.

Can I monitor Harbor without Prometheus?

Yes, you can use other tools like Datadog or New Relic, but Prometheus is the most common and integrates well with Kubernetes. Harbor also has a built-in health check endpoint.

How often should I scrape Harbor metrics?

Every 15-30 seconds is typical. More frequent scraping can add overhead, especially if you have many pods.

What are the most important Harbor metrics to watch?

Request latency, error rate, storage usage, and replication success rate. Also watch GC job duration and artifact count.

How do I get alerts to Slack?

Configure Alertmanager with a Slack receiver. In the kube-prometheus-stack, edit the alertmanager config to add your webhook URL.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →