Setting Up a Monitoring Stack in Docker Compose

A journey from a random guy to a random tech guy :D

Components

Grafana: The analytics and monitoring solution with support for multiple data sources, including Prometheus.

Prometheus: The monitoring and alerting toolkit, collecting metrics from configured targets at specified intervals.

Node Exporter: A Prometheus exporter for hardware and OS metrics exposed by *NIX kernels.

cAdvisor: Analyzes resource usage and performance characteristics of running containers.

Loki: A horizontally-scalable, highly-available, multi-tenant log aggregation system.

NGINX: Used as a reverse proxy to expose Grafana on the internet securely.

Configuration Files Overview

docker-compose.yml: Defines the services, networks, and volumes for the Docker containers.

prometheus.yml: Configuration file for Prometheus to define scrape targets and intervals.

grafana.ini: Configuration file for Grafana's settings, including SMTP for email notifications.

loki-config.yml: Configuration file for Loki to define the storage backend and other settings.

domain.conf: NGINX configuration for proxying requests to Grafana and handling WebSocket connections.

Installation Steps

Configure Docker Daemon: Enable Docker metrics by adding the following to /etc/docker/daemon.json and restarting Docker:

 {
   "metrics-addr": "127.0.0.1:9323"
 }
 # Then reload Docker configurations:
 sudo systemctl reload docker

Install Loki plugin for Docker: Install the Loki plugin for Docker to enable log collection:

 docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

Clone the Repository: Clone the repository to your host machine:

 git clone https://gist.github.com/398e94fb0aebb994b25223d99e7ae769.git monitoring-stack
 cd monitoring-stack

Start the Services: Start the monitoring stack using Docker Compose:

 docker-compose up -d

Access Grafana: Access Grafana at http://<your-domain> and log in with the default credentials (username: admin, password: admin).

Important Notes:

Replace placeholder values in grafana.ini with your SMTP credentials to enable email notifications.
Adjust the domain.conf to match your domain and specific requirements.
Review Docker and NGINX logs in case of any issues during setup.

How to send container logs to loki:

 docker run --name nginx-loki -d \
 -p 8080:80 \
 --log-driver=loki \
 --log-opt loki-url=http://localhost:3100/loki/api/v1/push \
 nginx:latest

Comments