Unexpected Memory Leak When Using asyncio.gather with Large Data in Python 3.10
I'm experiencing an unexpected memory leak when using `asyncio.gather` to handle multiple coroutines that process large data sets. My application processes CSV files and I use `pandas` to manipulate data, but when gathering results, the memory usage spikes significantly and does not seem to release after processing, which leads to eventual crashes due to memory exhaustion. Here's a simplified version of my code: ```python import asyncio import pandas as pd async def process_file(file_path): # Simulate a delay for reading a file await asyncio.sleep(1) df = pd.read_csv(file_path) # Perform some data manipulation df['new_column'] = df['existing_column'] * 2 return df async def main(file_paths): tasks = [asyncio.create_task(process_file(fp)) for fp in file_paths] results = await asyncio.gather(*tasks) return results file_list = ['file1.csv', 'file2.csv', 'file3.csv'] asyncio.run(main(file_list)) ``` I monitored the memory usage using `memory_profiler`, and it shows that after the `asyncio.gather`, the memory consumed by the process is significantly higher than expected and does not drop even after the function completes. I've tried explicitly deleting the `df` variable after returning it from `process_file`, but it didn't help. Here's what I tried: ```python async def process_file(file_path): await asyncio.sleep(1) df = pd.read_csv(file_path) df['new_column'] = df['existing_column'] * 2 result = df.copy() del df # Attempt to delete df return result ``` Despite this approach, the memory footprint remains high. I suspect this may be related to how `pandas` handles large DataFrames and their references in memory when returned from coroutines. Is there a better way to manage memory usage in this case? Are there best practices for using `asyncio` with libraries like `pandas` that anyone can suggest? Any insights would be greatly appreciated.