Monitoring AWS EC2 Instance using Grafana , Loki and Promtail

Requirements, AWS EC2 instance with Docker installed.

Promtail is the agent, responsible for gathering logs and sending them to Loki.

Loki is the main server, responsible for storing logs and processing queries.

Grafana is used for querying and displaying the logs.

-Install Grafana on Debian or Ubuntu

sudo apt-get install -y apt-transport-https

sudo apt-get install -y software-properties-common wget

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

Stable release

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

Beta release

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com beta main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

# Update the list of available packages

sudo apt-get update

# Install the latest OSS release:

sudo apt-get install grafana

Install Loki and Promtail using Docker

Download Loki Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

Run Loki Docker container

docker run -d --name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 --config.file=/mnt/config/loki-config.yaml

Download Promtail Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Promtail Docker container

docker run -d --name promtail -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.8.0 --config.file=/mnt/config/promtail-config.yaml

Using the instance public IP and port 3000 you should be able to access Grafana.

The default user name and password is admin and it ask you to change it before login.

Once logged in to Grafana, click add data source and select Loki as a data source and enter the IP of Loki with port 3100 and click save & continue.

Now, promtail gathers the logs and sends them to Loki, which is stored in /var/log/*log.

So, we need to add var logs as a label from where we can filter and check specific words or event. In the below we are searching varlogs folder for error.

Click run query and got to visualization logs and add it to the dashboard panel.