PowerShell 7.3 - Difficulty Filtering Large Log Files with Custom Date Range
I'm having trouble with I'm trying to figure out I'm trying to configure After trying multiple solutions online, I still can't figure this out... I'm trying to filter a large log file (approximately 1GB) in PowerShell 7.3 based on a custom date range, but I'm running into performance optimization and unexpected behavior. I want to extract entries between two specific dates. Here’s the code I've written so far: ```powershell $logFilePath = 'C:\Logs\application.log' $startDate = Get-Date '2023-10-01' $endDate = Get-Date '2023-10-31' Get-Content $logFilePath | ForEach-Object { if ($_ -match '\[(\d{4}-\d{2}-\d{2})') { $entryDate = [datetime]$matches[1] if ($entryDate -ge $startDate -and $entryDate -le $endDate) { $_ } } } ``` The log entries have timestamps in the format `[YYYY-MM-DD HH:MM:SS]`, but the filtering seems to take forever and often hangs at around 60% of the file. I also noticed that sometimes it skips entries that should be included, particularly when the date is at the boundaries. I've tried using `Select-String` combined with `Where-Object`, but it didn’t yield better performance either: ```powershell Select-String -Path $logFilePath -Pattern '\[(\d{4}-\d{2}-\d{2})' | Where-Object { $entryDate = [datetime]$matches[1] $entryDate -ge $startDate -and $entryDate -le $endDate } ``` However, this also resulted in a important amount of data being skipped. The behavior I get when using `Select-String` is `want to call method on a null-valued expression`. I've tried increasing the buffer size and adjusting `Get-Content` parameters like `-ReadCount`, but the results haven’t improved. What am I doing wrong, and how can I efficiently filter these log entries within the specified date range without running into performance optimization? Any help would be greatly appreciated! Has anyone else encountered this? I'm working on a microservice that needs to handle this. Thanks for your help in advance! I'm using Powershell 3.9 in this project. Any help would be greatly appreciated! This is part of a larger microservice I'm building. Has anyone dealt with something similar?