Grafana Loki previous logs lost on docker container remove and then run

Question

I am losing my logs everytime I remove and then start Grafana Loki and promtail containers (i.e. all the logs till the time of container removal). I want to know if its possible to retain logs on local filesystem between container removal and then start again.

Here is my docker launch command for loki:

docker run --name loki -d --restart unless-stopped -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.9.1 -config.file=/mnt/config/loki-config.yaml

And here is my loki-config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb:
    directory: /loki/index
  
compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true
  
chunk_store_config:
  max_look_back_period: 48h
  
limits_config:
  retention_period: 48h
  
table_manager:
  retention_deletes_enabled: true
  retention_period: 48h

ruler:
  alertmanager_url: http://localhost:9093

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
analytics:
 reporting_enabled: false

Answer 1

  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules

You store stuff in /tmp/loki. You have to preserve /tmp/loki directory between executions.

docker run -v /some/persistent/place:/tmp/loki ....

Consider renaming it just /loki, not really a /tmp anymore.

Answer 2

Instead of using the local directory create a docker volume and use that volume in your command:

create volumes for Loki:

docker volume create loki-data

Your command for docker run should be like:

docker run --name loki -d --restart unless-stopped -v loki-data:/mnt/config -p 3100:3100 grafana/loki:2.9.1 -config.file=/mnt/config/loki-config.yaml

to copy your volume data to the local system or viceversa then explore docker cp command.

docker cp loki:/mnt/config /path/to/local/backup