How to aggregate logs from multiple microservices using Fluentd and Elasticsearch in a Kubernetes cluster?
This might be a silly question, but Hey everyone, I'm running into an issue that's driving me crazy. I am trying to set up a centralized logging system for my microservices architecture running in a Kubernetes cluster. I've been using Fluentd to collect logs from various services, and I want to aggregate these logs in Elasticsearch. I configured Fluentd using a ConfigMap, but I keep working with an scenario where logs from one of my services are missing in Elasticsearch. Here’s a snippet of my Fluentd configuration in the ConfigMap: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config data: fluent.conf: | <source> @type kubernetes @id kubernetes @include_metadata true <storage> @type local persistent true </storage> </source> <filter **> @type record_transformer <record> cluster_name ${K8S_CLUSTER_NAME} </record> </filter> <match **> @type elasticsearch host elasticsearch-client port 9200 logstash_format true include_tag_key true tag_key @log_name index_name fluentd_${tag} </match> ``` I have verified that logs are being produced by the microservices and that Fluentd is correctly picking up logs from most services. However, for one service, I see no errors in the Fluentd logs, but the logs are simply not appearing in Elasticsearch. I tried increasing the verbosity of the Fluentd logs, but still, there's no indication of the logs being processed. I also checked the permissions for the service account that Fluentd is using, and they seem fine. The logs for the problematic service contain a lot of JSON objects, each logging various levels of information (INFO, behavior, DEBUG). Has anyone faced a similar scenario, and how can I debug or resolve this missing log question? Any insights or tips on configuration would be greatly appreciated! What would be the recommended way to handle this? I'm working on a REST API that needs to handle this. Is this even possible?