👀 Views: 82 💬 Answers: 1 📅 Created: 2025-09-06

gcp bigquery python performance Python

I'm trying to implement I'm converting an old project and I'm facing significant performance issues when extracting data from BigQuery using the Python client library... My use case involves querying a large dataset (approximately 10 million rows) and writing the results to a CSV file for further processing. The query itself is optimized, but the extraction process seems to be taking an excessively long time. I’m using the `google-cloud-bigquery` library, version 2.34.0, and I'm attempting to retrieve the data using the `to_dataframe()` method. Here's a snippet of my code: ```python from google.cloud import bigquery import pandas as pd client = bigquery.Client() query = """ SELECT * FROM `my_project.my_dataset.my_table` WHERE some_condition = true """ # This is where I suspect the issue lies start_time = time.time() df = client.query(query).to_dataframe() # This takes too long end_time = time.time() print(f"Query execution time: {end_time - start_time} seconds") # Writing to CSV output_file = 'output.csv' df.to_csv(output_file, index=False) ``` Although the query returns results quickly when run directly in the BigQuery console, the data extraction through the client library can take upwards of 10 minutes. I've tried increasing the `max_results` parameter in the query options and even tested with smaller result sets, but the behavior remains consistent. I also checked my network bandwidth and found it to be stable. Is there a more efficient way to handle large data extractions from BigQuery using the Python client? Are there specific configurations or best practices I could apply to improve performance in this situation? Any insights would be greatly appreciated! I'd be grateful for any help. My team is using Python for this desktop app. Thanks in advance!