Grafana's Loki and Promtail Setup

This explains how to set up Loki and then configure Promtail to forward logs to Loki so that Grafana can read the logs.

Loki

Downloads can be found here: Loki Installation Methods

Docker-compose

The docker-compose file.

version: "3"

services:
  loki:
    image: grafana/loki:2.4.0
    volumes:
      - ./loki:/etc/loki
    restart: unless-stopped
    command: -config.file=/etc/loki/loki-config.yml
    ports:
      - '3100:3100'

The config file for Loki config-loki.yml.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Promtail

Downloads can be found here: Promtail Github Downloads

While there’s mulitple ways to install Promtail, this will cover Docker Compose and locally installed.

Docker Compose

Within your docker folder, create the promtail folder and then another promtail folder to house the config file.

mkdir -p promtail\promtail
cd promtail
touch docker-compose.yml
cd promtail
touch promtail-config.yml

version: "3"
services:
  promtail:
    image: grafana/promtail:2.4.0
    volumes:
      - /var/log:/var/log
      - ./promtail:/etc/promtail
    #ports:
     # - "1514:1514" # this is only needed if you are going to send syslogs
    restart: unless-stopped
    command: -config.file=/etc/promtail/promtail-config.yml

The config file. Be sure to change the client URL before implementing.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://[ip_address|localhost]:3100/loki/api/v1/push

scrape_configs:

# local machine logs

- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log
      host: __hostname__
- job_name: local2
  static_configs:
  - targets:
      - localhost
    labels:
      job: logs
      __path__: /var/log/messages 
      host: __hostname__

Run docker compose

docker-compose up -d

Local Install

The commands below are ran in sudo su -.

Download the file, unzip, change the permisisons and place it in a executable location.

curl -O -L "https://github.com/grafana/loki/releases/download/v2.5.0/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64
cp promtail-linux-amd64 /usr/local/bin

Create the config file to be used by Promtail. If Promtail is being installed on a machine that doesn’t have Loki installed, be sure to change - url: 'http://localhost:3100/loki/api/v1/push' to - url: 'http://loki_machine_ip:3100/loki/api/v1/push'

cat << EOF > /usr/local/bin/config-promtail.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: 'http://localhost:3100/loki/api/v1/push'

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/*log
          host: __computer_name__

  - job_name: system-main
    static_configs:
      - targets:
          - localhost
        labels:
          job: main
          __path__: /var/log/{messages,secure}
          host: __computer_name__
EOF

Create the promtail user. Depending on the log files, by default Promtail won’t be able to read certain log files due to permissions. The last 4 steps help with that. This makes it so that Promtail isn’t being ran as a root user.

adduser --system promtail
cd /var
setfacl -R -m u:promtail:rX log
chown promtail:promtail /tmp/positions.yaml
usermod -a -G systemd-journal promtail

By default, Promtail will open ports up. Use firewall rules to block unwanted access. The –zone flag is optional depending on what one’s setup is.

firewall-cmd --add-port={9080,9097}/tcp --zone=trusted --permanent
firewall-cmd --reload

Create the service file so that Promtail can be ran as a system service.

cat << EOF > /etc/systemd/system/promtail.service
[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=promtail
ExecStart=/usr/local/bin/promtail-linux-amd64 -config.file /usr/local/bin/config-promtail.yml

[Install]
WantedBy=multi-user.target
EOF

Start and enable the service.

systemctl start promtail
systemctl enable promtail.service
systemctl status promtail.service