👀 Views: 288 💬 Answers: 1 📅 Created: 2025-06-20

gcp bigquery python google-cloud data-ingestion Python

Does anyone know how to I'm using the Google Cloud Python client library to ingest data into BigQuery, but I'm running into a 'Quota Exceeded' behavior almost every time I try to load more than 1GB of data. Here's the code snippet I'm using for the ingestion: ```python from google.cloud import bigquery client = bigquery.Client() dataset_id = 'my_dataset' table_id = 'my_table' job_config = bigquery.LoadJobConfig( source_format=bigquery.SourceFormat.NEWLINE_DELIMITED_JSON, ) with open('data.json', 'rb') as source_file: job = client.load_table_from_file( source_file, f'{dataset_id}.{table_id}', job_config=job_config, ) job.result() # Waits for the job to complete. print('Loaded {} rows into {}:{}.'.format(job.output_rows, dataset_id, table_id)) ``` I initially thought this might be due to the dataset's daily quota limit, but I checked and it appears that I should still have capacity left. I'm using the `bigquery` library version 2.30.0, and my project is configured correctly with sufficient permissions. I also tried splitting the data into smaller chunks and loading them in a loop, but that doesn't seem to help either. The behavior message I'm working with is: ``` google.api_core.exceptions.Forbidden: 403 Quota exceeded for project 'my_project_id' (resource 'bigquery.googleapis.com') ``` Is there a specific limit I might be overlooking, or is it possible that there's an scenario with my project settings? Any insights would be greatly appreciated! Any pointers in the right direction? I'm using Python 3.11 in this project. Any pointers in the right direction?