Memory Leak in Python 3.10 when using SQLAlchemy with large result sets
I've been struggling with this for a few days now and could really use some help... I'm working with a memory leak when querying large datasets with SQLAlchemy in Python 3.10. My application retrieves data from a PostgreSQL database, and when I query a table with several million rows, the memory usage keeps increasing, eventually causing the application to crash. I've tried using the `yield_per()` method to load results in chunks, but it doesn't seem to solve the scenario. Here's a simplified version of my code: ```python from sqlalchemy import create_engine, select from sqlalchemy.orm import sessionmaker from mymodels import MyModel engine = create_engine('postgresql://user:password@localhost/mydatabase') Session = sessionmaker(bind=engine) with Session() as session: query = select(MyModel) for instance in session.execute(query).scalars().yield_per(1000): # Process instance print(instance) ``` I also tried using `session.close()` after processing each batch, but the memory usage still doesn't stabilize. The application can run for a few hours, but then it consumes all available memory. I've read that garbage collection in Python can sometimes lead to delayed memory release. Is there a specific way to enforce garbage collection or manage memory better when working with SQLAlchemy? Any suggestions would be greatly appreciated! For context: I'm using Python on Linux.