告别笨重的 ELK，试试用 Loki 处理日志吧！

更新于2024-05-24：如果你没有查找历史日志的需求，只是想简单地看看实时日志。docker 的 Coolify 和 Kubernetes 的 Lens 自带的日志浏览功能就够用了。

日志领域主流解决方案是 elk，但是这一套部署复杂、资源占用高，不适合个人开发者使用，如果你需要一个比命令行漂亮一些的日志查看工具，loki 就非常合适了。

promtail从 k8s/docker 中收集容器标准输出日志，发送到 loki 存储，grafana 在网页端对日志进行搜索和展示。非常轻量级，200MB 左右的内存就可以搞定。

安装

首先在一个安装了 docker 的服务器部署 loki、grafana 和收集本机 docker 容器日志的promtail：

version: '3'
services:
  grafana:
    image: grafana/grafana-enterprise
    container_name: grafana
    restart: unless-stopped
    networks:
      - npm
      - grafana
    # ports:
    #   - 3000:3000
    volumes:
      - ./volume/grafana/data/:/var/lib/grafana
  loki:
    image: grafana/loki
    container_name: loki
    restart: unless-stopped
    volumes:
      - ./volume/loki/config/:/etc/loki/
      - ./volume/loki/data/:/loki/
    # 修改loki默认配置文件路径
    command: -config.file=/etc/loki/loki.yml
    # ports:
    #   - 3100:3100
    networks:
      - npm
      - grafana
  promtail:
    image:  grafana/promtail:latest
    container_name: promtail
    restart: unless-stopped
    volumes:
      - ./volume/promtail/config/promtail.yml:/etc/promtail/docker-config.yaml
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/docker-config.yaml
    networks:
      - grafana
      
networks:
  npm:
    external: true
  grafana:

这里出现了两个 docker 网络：npm 和 grafana，npm（nginx proxy manager） 是为了反向代理，grafana 是为了让这三个容器之间互通。

需要用到的 loki 配置文件：

auth_enabled: false

server:
  http_listen_port: 3100

common:
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://alertmanager:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
analytics:
 reporting_enabled: false

# 允许接收旧日志
limits_config:
  reject_old_samples: false

promtail 配置文件：

# https://grafana.com/docs/loki/latest/clients/promtail/configuration/
# https://docs.docker.com/engine/api/v1.41/#operation/ContainerList
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push
scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: 'logstream'
      - source_labels: ['__meta_docker_container_label_logging_jobname']
        target_label: 'job'

这个 promtail 收集 docker 中所有容器的标准输出日志，并打上“container=容器名”的 lable 发送给 loki，以区分不同容器的日志。

安装好 docker 这一套以后，登录 grafana，添加 loki 数据源，在 explore 页面可以通过“{container=容器名}”的查询语句查看 docker 容器的日志。

k8s 的日志如何推送到 loki？

我在另外一台安装了 k8s 的服务器运行后端，我们还要把 k8s pod 的日志发送到 loki 里，便于统一查看。

--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: grafana/promtail
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env: 
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: https://你的 loki 地址/loki/api/v1/push
      basic_auth:
        username: 你的 loki 用户名
        password: 你的 loki 密码

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: default
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

这会在你k8s 的每一个节点上运行一个 promtail，收集pod日志发送到loki。

然后到 grafana 导入这个 dashboard，你就可以方便地查看 pod 日志了。

但是细心的同学应该能发现这个 crd 里有一些我没提到的参数（https 的 loki url，用户名，密码）。

这是因为我们不想让日志暴露在全世界面前，我们可以通过 npm 反向代理安全、便捷地实现这一点。

我的另一篇文章：轻松配置 https：Let‘s Encrypt 介绍及 Nginx Proxy Manager 实用操作教程 介绍了如何给 http 服务加上反向代理，并使用 basic auth 来保护接口。然后你就能得到你 loki 的 url、用户名和密码了。