Optimizing Excel data import for machine learning model performance
Quick question that's been bugging me - I'm relatively new to this, so bear with me... Currently developing a machine learning model that pulls data from an Excel file. The file contains multiple sheets with varying structures, and I'm trying to optimize the data import process since the current method is starting to slow down my model training. I started with a simple approach using `pandas` to read the sheets: ```python import pandas as pd # Read multiple sheets xls = pd.ExcelFile('data.xlsx') # Load all sheets into a dictionary of DataFrames sheet_dict = {sheet_name: xls.parse(sheet_name) for sheet_name in xls.sheet_names} ``` While this works, I've noticed that the loading time increases significantly with larger datasets, especially when sheets have complex formulas or lots of columns. I considered using the `usecols` parameter to limit the data being read: ```python # Loading specific columns from a sheet sheet_data = pd.read_excel('data.xlsx', sheet_name='Sheet1', usecols='A:C,F') ``` This does help, but I still feel like there's room for improvement. Has anyone implemented a more efficient way to handle this situation? Additionally, I've been reading about using `pyxlsb` for binary Excel files to speed up the process, but Iām unsure if it would significantly impact performance. Another potential solution Iām exploring is converting the Excel files into a more suitable format, like CSV or Parquet, but Iām worried about losing the formulas and formatting that can be useful for data analysis. Lastly, I've also set up Excel's ability to 'Refresh Data' for queries, but automating that with Python to ensure I always have the latest data is proving to be another hurdle. If anyone has faced similar challenges or has best practices for optimizing Excel data imports in machine learning contexts, I'd greatly appreciate your insights. For context: I'm using Python on Linux. Is this even possible? Has anyone dealt with something similar?