Linux service stops logging

Question

I have a service running. The service invokes a long running python script which "prints" some status messages. I can see these status messages with journalctl -u myservice.service. But after about 6 hours I am no longer able to see new outputs from the service.

If I run systemctl status myservice.service. I can tell that service is still running.

I suspected it had to do something with the logging, so I reduced the time intervall of the log output from once per minute to once per 10 seconds. Fair enough, the problem still occured but after just 1 hour. Thus it seems as if I have an issue with log space or log rotation. Can someone assist me with how to solve this?

I started to look into the journald.conf file and the content looks like this.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
#SystemMaxUse=
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

I tried changing the content as below and then ran sudo systemctl restart systemd-journald.service without any success.

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=1G            # Increase to allow more disk space for logging
#SystemKeepFree=
#SystemMaxFileSize=
#RuntimeMaxUse=
#RuntimeKeepFree=
#RuntimeMaxFileSize=
MaxRetentionSec=1month     # Retain journal files for up to one month
MaxFileSec=1week           # Keep individual journal files for up to one week
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48K

Any ideas?

Answer 1

I had a similar problem where my service log where suppressed due to the amount of messages received inside the RateLimitInterval window. But if it's always exactly 6 hours... quite weird.
Could it be that there are RateLimitInterval / RateLimitBurst misconfigured somewhere? Did you notice any suppression message inside your syslog/journal? Something like systemd-journal[xxxx]: Suppressed <number> messages from <your_service>.?
It's weird, because I think your default values for those parameters are good and don't match your issue... Maybe your service provides rate limits for itself through LogRateLimitIntervalSec= and/or LogRateLimitBurst= ? Because in such case, those values override the settings specified in journald conf.
In case you see the suppression message, you could try to set

RateLimitInterval=0s
RateLimitBurst=0

to fully disabled them (just for debugging: then you should tune it), and maybe you can also increase the size of

RuntimeMaxUse

and, in case you think that the default one eighth of the values configured with SystemMaxUse and RuntimeMaxUse isn't enough, you can also increase

SystemMaxFileSize
RuntimeMaxFileSize

(ref. https://www.freedesktop.org/software/systemd/man/latest/journald.conf.html)