OCI Data Science: working with 'Resource Limit Exceeded' scenarios When Running Jupyter Notebooks with Large Datasets
Quick question that's been bugging me - I'm performance testing and I'm confused about I keep running into I'm currently working on a data science project using Oracle Cloud Infrastructure (OCI) Data Science service. I have a Jupyter notebook where I'm trying to process a large dataset (around 15GB) using Pandas. However, I frequently encounter a 'Resource Limit Exceeded' behavior when attempting to read the dataset into a DataFrame with the following code: ```python import pandas as pd # Attempt to load a large CSV file dataset_path = '/path/to/large_dataset.csv' df = pd.read_csv(dataset_path) ``` I've ensured that my compute instance is of shape VM.Standard2.4 and has 32GB of RAM, which should be sufficient for handling datasets of this size. I've also tried increasing the shape to VM.Standard2.8, but the scenario continues. To troubleshoot, I monitored the resource usage and found that memory utilization spikes significantly just before the behavior occurs. I’ve also experimented with chunking the data by using the `chunksize` parameter in the `read_csv` method: ```python for chunk in pd.read_csv(dataset_path, chunksize=100000): process_chunk(chunk) ``` Even with chunk processing, I still experience memory issues as the process overall seems to exceed the allocated memory limits. I’ve checked the OCI console for any additional quotas or limits that might be affecting performance but haven’t found anything unusual. Is there a recommended way to handle large datasets in OCI Data Science that can help avoid these resource limit errors? Any best practices or configurations I should consider? Also, are there specific configurations in the Jupyter environment that could improve performance? My development environment is Linux. What am I doing wrong? I'm working on a web app that needs to handle this. I'm developing on CentOS with Python. Is this even possible? For context: I'm using Python on Windows 11. I'm working on a application that needs to handle this. What would be the recommended way to handle this?