High CPU Usage from `systemd-journald` on Fedora 38 During High Log Volume
I'm stuck trying to I'm experiencing unusually high CPU usage from `systemd-journald` on my Fedora 38 server during peak log generation times. Normally, I would expect some load, but it's consistently hitting 90% CPU usage when my application generates extensive logs due to user activity. I attempted to limit the maximum size of the journal logs to reduce the load. My `/etc/systemd/journald.conf` has the following configuration: ```ini [Journal] SystemMaxUse=100M SystemKeepFree=50M MaxFileSec=1h MaxRetentionSec=7d ``` Despite this configuration, the CPU usage remains high and the logs indicate messages like: ``` -- Logs begin at Mon 2023-10-23 10:00:00 UTC, end at Mon 2023-10-23 12:00:00 UTC. -- Oct 23 12:00:00 myserver systemd-journald[1234]: Journal started Oct 23 12:00:00 myserver systemd-journald[1234]: Suppressed 1000 messages from myapp ``` I also tried adjusting the `SyslogLevel` and `MaxLevelStore`, but that didn't seem to help: ```ini [Journal] SyslogLevel=info MaxLevelStore=notice ``` As a workaround, I implemented a log rotation script that compresses old logs, but I would prefer a more elegant solution. I suspect that the high log volume might be overwhelming the journald service itself. Has anyone encountered a similar scenario and found a solution to effectively manage log volume without losing important information or experiencing high CPU usage? Is this even possible?