How to handle excessive log noise in a Node.js Express application with Winston and Elasticsearch?
After trying multiple solutions online, I still can't figure this out. I'm currently working with a scenario with log management in my Node.js Express application that uses Winston for logging and Elasticsearch for log storage. My application is quite chatty and generates a lot of logs, including information logs for every request, which makes it hard to sift through them for meaningful insights. I have the following logging setup: ```javascript const express = require('express'); const winston = require('winston'); const { createLogger, format, transports } = winston; const app = express(); const logger = createLogger({ level: 'info', format: format.combine( format.timestamp(), format.json() ), transports: [ new transports.Console(), new transports.Elasticsearch({ level: 'info', index: 'logs', // Elasticsearch configuration here }) ] }); app.use((req, res, next) => { logger.info(`Request: ${req.method} ${req.url}`); next(); }); app.get('/', (req, res) => { res.send('Hello World!'); }); app.listen(3000, () => { logger.info('Server is running on port 3000'); }); ``` While this setup works, I noticed that it produces a important amount of logs, particularly for less important requests. In my Elasticsearch instance, I see many similar entries cluttering the logs: ``` { "timestamp": "2023-10-01T12:00:00Z", "level": "info", "message": "Request: GET /" } ``` My main scenario is that I'm unable to filter out less critical logs while still keeping track of important ones, especially in production. I tried adjusting the logging level to 'warn', but that also removed too much valuable information. Iām looking for best practices on how to manage log verbosity in such applications. Is there a way to categorize logs or to implement log sampling? Additionally, if I could highlight certain routes or errors that should always be logged regardless of the configuration, that would be ideal. Any advice or code snippets would be greatly appreciated! My development environment is macOS. Any ideas what could be causing this?