PHP 8.2 - Difficulty Handling Large CSV Imports with Memory Limit Exceeded scenarios
I've looked through the documentation and I'm still confused about I'm currently working with PHP 8.2 and trying to import a large CSV file (over 1GB) into my MySQL database using the built-in `fgetcsv()` function... However, I'm running into a `Fatal behavior: Allowed memory size of 134217728 bytes exhausted` behavior after processing a certain number of rows. To mitigate memory usage, I tried increasing the memory limit in my `php.ini` file to 512M with `memory_limit = 512M`, but I still encounter this scenario. My current implementation looks like this: ```php $filepath = 'path/to/large_file.csv'; if (($handle = fopen($filepath, 'r')) !== false) { while (($data = fgetcsv($handle, 1000, ',')) !== false) { // Process the row and insert it into the database $stmt = $pdo->prepare("INSERT INTO my_table (col1, col2) VALUES (?, ?)"); $stmt->execute([$data[0], $data[1]]); } fclose($handle); } else { echo 'Could not open file.'; } ``` I considered using `fgetcsv()` with a loop to handle smaller chunks, but I need to seem to find a way to keep track of where I left off if the script times out or fails. Additionally, I've tried using the `LIMIT` clause in SQL, but that doesn't apply directly to the CSV import process. What's the best approach to handle such a large CSV import without running out of memory? Are there any specific libraries or methods that can help manage memory consumption more effectively? Any suggestions or examples would be greatly appreciated! My development environment is Ubuntu. Any help would be greatly appreciated! For reference, this is a production web app. Hoping someone can shed some light on this. This issue appeared after updating to Php LTS.