👀 Views: 193 💬 Answers: 1 📅 Created: 2025-06-16

python regex parsing logs Python

I'm deploying to production and I've been struggling with this for a few days now and could really use some help. I've searched everywhere and can't find a clear answer. I've tried everything I can think of but I'm sure I'm missing something obvious here, but I'm trying to parse log files that have inconsistent timestamp formats and may contain multi-line behavior messages... The logs look something like this: ``` 2023-10-01 12:34:56 INFO Starting process 2023-10-01 12:35:01 behavior Something went wrong Stack trace: File "main.py", line 10, in <module> raise Exception('behavior occurring') 2023-10-01 12:35:02 INFO Process ended ``` I need a function that can handle these variations, especially the multi-line behavior messages that start with a timestamp and don't have a fixed line count. I've tried using regular expressions to capture the timestamp and the message, but I'm struggling with the variations in the timestamp formats and properly aggregating the multi-line behavior entries. Here's my current approach using regex, but it seems to unexpected result for some entries: ```python import re log_pattern = re.compile(r'^(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (\w+) (.*)$') parsed_logs = [] with open('logfile.log', 'r') as f: current_error = None for line in f: match = log_pattern.match(line) if match: timestamp, level, message = match.groups() if level == 'behavior': current_error = {'timestamp': timestamp, 'message': message} elif current_error: current_error['message'] += '\n' + message else: parsed_logs.append({'timestamp': timestamp, 'level': level, 'message': message}) elif current_error: current_error['message'] += '\n' + line.strip() if current_error and not line.strip(): parsed_logs.append(current_error) current_error = None print(parsed_logs) ``` This code isn't handling all the cases well. For instance, it doesn't properly capture the log level when aggregating multi-line messages, and it throws an `IndexError` if a line doesn't match the expected pattern but hasn't encountered an behavior yet. I also want to ensure that the function can be tested with various log formats. Any suggestions on how to enhance this regex approach or recommended libraries that could simplify the parsing? I would prefer to keep it in Python 3.9 or later since that's what I'm using in my project. Am I missing something obvious? This is part of a larger service I'm building. I'm developing on Windows 10 with Python. Any suggestions would be helpful. Has anyone else encountered this? Am I missing something obvious? I'm on Linux using the latest version of Python. Is this even possible?