👀 Views: 64 💬 Answers: 1 📅 Created: 2025-06-16

python regex file-parsing Python

I'm maintaining legacy code that This might be a silly question, but I'm relatively new to this, so bear with me. I'm working on parsing custom log files generated by our application, which have a specific format like this: ``` 2023-10-01 12:00:00 INFO User JohnDoe logged in 2023-10-01 12:05:00 ERROR File not found: config.yaml 2023-10-01 12:10:00 INFO User JohnDoe logged out ``` I’m using Python 3.10 and the `logging` module to handle the parsing. I've created a custom parser function to read these logs line by line, but I'm running into issues where the memory usage spikes significantly, especially with larger log files, and occasionally I get missing entries in my parsed output. Here's an example of the code I have: ```python import re class LogParser: def __init__(self, filename): self.filename = filename self.entries = [] def parse(self): with open(self.filename, 'r') as file: for line in file: match = re.match(r'^(\S+) (\S+) (\w+) (.*)$', line) if match: timestamp, log_level, user_action = match.groups()[:3] self.entries.append({ 'timestamp': timestamp, 'log_level': log_level, 'message': user_action }) return self.entries ``` When I run this on a log file with about 10,000 entries, I see that the memory consumption is significantly higher than expected. Additionally, when I check the output, sometimes I notice that not all log entries are appearing in the final list, particularly those with the 'ERROR' log level. I've tried optimizing the read process by switching to a buffered approach and limiting the number of entries stored, but I still experience high memory usage. For example, using a generator instead of a list: ```python def parse(self): with open(self.filename, 'r') as file: for line in file: match = re.match(r'^(\S+) (\S+) (\w+) (.*)$', line) if match: yield { 'timestamp': match.group(1), 'log_level': match.group(2), 'message': match.group(3) } ``` Still, the memory issue persists, and I haven't been able to figure out why some entries are missing. Any advice on how to efficiently parse these logs while keeping memory usage low and ensuring that all entries are captured? Also, are there any common pitfalls I should be aware of when using regex for this kind of task? Is there a better approach? This is happening in both development and production on Debian. Thanks for your help in advance!