Setting Up a Monitoring Stack in Docker Compose

Using Grafana, Prometheus, Node Exporter, cAdvisor, and Loki, Exposed via NGINX

This guide provides instructions on setting up a comprehensive monitoring stack using Grafana, Prometheus, Node Exporter, cAdvisor and Loki. These components are orchestrated with Docker Compose and exposed via an NGINX reverse proxy, making them accessible through a single domain.

Components

  • Grafana: The analytics and monitoring solution with support for multiple data sources, including Prometheus.

  • Prometheus: The monitoring and alerting toolkit, collecting metrics from configured targets at specified intervals.

  • Node Exporter: A Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.

  • cAdvisor: Analyzes resource usage and performance characteristics of running containers.

  • Loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system.

  • NGINX: Used as a reverse proxy to expose Grafana on the internet securely.

Prerequisites

  • Docker and Docker Compose installed on your host machine.

  • Domain name configured to point to the host machine (for NGINX configuration).

Configuration Files Overview

  • docker-compose.yml: Defines the services, networks, and volumes for the Docker containers.

  • prometheus.yml: Configuration file for Prometheus to define scrape targets and intervals.

  • grafana.ini: Configuration file for Grafana's settings, including SMTP for email notifications.

  • loki-config.yml: Configuration file for Loki to define the storage backend and other settings.

  • domain.conf: NGINX configuration for proxying requests to Grafana and handling WebSocket connections.

Installation Steps

  1. Configure Docker Daemon: Enable Docker metrics by adding the following to /etc/docker/daemon.json and restarting Docker:

     {
       "metrics-addr": "127.0.0.1:9323"
     }
     # Then reload Docker configurations:
     sudo systemctl reload docker
    
  2. Install Loki plugin for Docker: Install the Loki plugin for Docker to enable log collection:

     docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
    
  3. Clone the Repository: Clone the repository to your host machine:

     git clone https://gist.github.com/398e94fb0aebb994b25223d99e7ae769.git monitoring-stack
     cd monitoring-stack
    
  4. Start the Services: Start the monitoring stack using Docker Compose:

     docker-compose up -d
    
  5. Access Grafana: Access Grafana at http://<your-domain> and log in with the default credentials (username: admin, password: admin).

Important Notes:

  • Replace placeholder values in grafana.ini with your SMTP credentials to enable email notifications.

  • Adjust the domain.conf to match your domain and specific requirements.

  • Review Docker and NGINX logs in case of any issues during setup.

  1. How to send container logs to loki:

     docker run --name nginx-loki -d \
     -p 8080:80 \
     --log-driver=loki \
     --log-opt loki-url=http://localhost:3100/loki/api/v1/push \
     nginx:latest