Monitor a Server with Prometheus and Grafana

This guide walks you through setting up Prometheus to collect server metrics (CPU, memory, disk, network) and Grafana to visualize them on dashboards. You’ll end with a live monitoring stack that alerts you when something goes wrong. You’ll need a Linux server (Ubuntu/Debian used here), sudo access, and a basic comfort with the command line.

What You’ll Need

• A Linux server (physical or VM) with sudo privileges.
• SSH access to that server.
• Ports 9090 (Prometheus) and 3000 (Grafana) reachable (open in firewall if applicable).
• Basic familiarity with editing files in a terminal (nano or vim).

Step-by-Step

1. Update the system
   SSH into your server and run:
   sudo apt update && sudo apt upgrade -y
2. Create a dedicated system user for Prometheus
   Run:
   sudo useradd --no-create-home --shell /bin/false prometheus
3. Download Prometheus
   Check the latest version at prometheus.io/download. As of writing, 2.53.0 is current. Replace the version in the commands below if newer.
   cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
4. Extract and install binaries
   tar xvf prometheus-2.53.0.linux-amd64.tar.gz
sudo cp prometheus-2.53.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.53.0.linux-amd64/promtool /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
5. Set up Prometheus directories and config
   sudo mkdir -p /etc/prometheus /var/lib/prometheus
sudo cp -r prometheus-2.53.0.linux-amd64/consoles /etc/prometheus
sudo cp -r prometheus-2.53.0.linux-amd64/console_libraries /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
6. Create the Prometheus configuration file
   Edit /etc/prometheus/prometheus.yml with:

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
         - targets: ['localhost:9090']
   
     - job_name: 'node'
       static_configs:
         - targets: ['localhost:9100']

7. Create a systemd service for Prometheus
   Create /etc/systemd/system/prometheus.service with:

   [Unit]
   Description=Prometheus
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=prometheus
   Group=prometheus
   Type=simple
   Restart=always
   ExecStart=/usr/local/bin/prometheus \
     --config.file=/etc/prometheus/prometheus.yml \
     --storage.tsdb.path=/var/lib/prometheus/

8. Start and enable Prometheus
   sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
9. Install Node Exporter
   Node Exporter provides system metrics. Download and install it similarly:
   cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvf node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
10. Create a service for Node Exporter
    Create /etc/systemd/system/node_exporter.service with:

    [Unit]
    Description=Node Exporter
    After=network.target
    
    [Service]
    User=node_exporter
    Group=node_exporter
    Type=simple
    ExecStart=/usr/local/bin/node_exporter
    
    [Install]
    WantedBy=multi-user.target

    Then start it:
    sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

11. Verify Prometheus is scraping
    Visit http://your-server-ip:9090/targets in a browser. You should see both prometheus and node targets with status UP.
12. Install Grafana
    Add the Grafana repository and install:
    sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana
13. Start Grafana
    sudo systemctl start grafana-server
sudo systemctl enable grafana-server
14. Log into Grafana and add Prometheus as a data source
    Open http://your-server-ip:3000, log in with admin/admin (you’ll be prompted to change). Go to Configuration > Data Sources > Add data source, choose Prometheus, and set URL to http://localhost:9090. Click Save & Test.
15. Import a dashboard
    Go to Dashboards > Import, enter ID 1860 (a popular Node Exporter dashboard), click Load, select your Prometheus data source, and Import. You’ll now see a full server dashboard.
16. Set up alerts (optional)
    In Grafana, go to Alerting > Alert rules to create rules like CPU > 80% for 5 minutes. You can add notification channels (email, Slack) under Contact points.

Common Mistakes

• Firewall blocking ports: Forgetting to open 9090 and 3000. Use sudo ufw allow 9090,3000/tcp if using UFW.
• Wrong file permissions: Prometheus and Node Exporter services fail if binaries aren’t owned by their users. Double-check with ls -l /usr/local/bin/prometheus.
• Typo in config YAML: YAML is strict about spaces. Use promtool check config /etc/prometheus/prometheus.yml to validate.
• Scraping localhost from a remote server: If Prometheus is on a different host than Node Exporter, change target to the exporter’s IP.

Tips

• Use a reverse proxy (like Nginx) to add HTTPS to Grafana for production.
• Set up retention: add --storage.tsdb.retention.time=30d to Prometheus service to limit disk usage.
• Back up /var/lib/prometheus for historical data.
• Use Grafana’s provisioned dashboards to version-control your setup.

Troubleshooting

• Prometheus service won’t start: Check logs with journalctl -u prometheus -n 50. Usually a config error or permission issue.
• Targets show DOWN: From the server, run curl localhost:9100/metrics. If it fails, Node Exporter isn’t running. Check systemctl status node_exporter.
• Grafana can’t connect to Prometheus: Ensure Prometheus is listening on localhost:9090. Test with curl localhost:9090. If Grafana runs in a container, use host.docker.internal instead of localhost.
• Dashboard shows no data: Verify the data source is correct and that the dashboard’s queries match your metrics. Try a simple query like up in Explore.
• Ports already in use: Change ports in the service files and update configs accordingly.

Frequently Asked Questions

Do I need a separate server for Prometheus and Grafana?

No, you can run both on the same machine. For larger setups, separate them to avoid resource contention.

Can I monitor multiple servers?

Yes. Install Node Exporter on each server and add them as targets in Prometheus config under the same job, or use file-based discovery.

How do I secure Prometheus and Grafana?

Put them behind a reverse proxy with TLS, and use Grafana’s built-in authentication. Prometheus itself has no auth; use a firewall to restrict access.

What metrics does Node Exporter provide?

CPU, memory, disk I/O, filesystem usage, network traffic, and more. All under the node_ prefix.

Can I use something other than Node Exporter?

Yes, there are exporters for MySQL, PostgreSQL, Nginx, and many more. The setup is similar: run the exporter and add a scrape job.

How do I update Prometheus?

Download the new version, replace the binary, and restart the service. Your data and config remain intact.

Related

• Waymo Robotaxi Subscription: How It Works
• NetworkManager Dispatcher Scripts: Can They Secure Public Wi-Fi?
• Moto G200 with Snapdragon 888+ and 108MP Cameras: Launched?