GCP BigQuery Ingestion Latency with Streaming Inserts in Go
I tried several approaches but none seem to work. I'm experiencing important latency issues when using streaming inserts to load data into BigQuery tables from my Go application... Despite setting the batch size to 1000 rows and using the latest `cloud.google.com/go/bigquery` version 1.25.0, the insert operations are taking longer than expected. I expected near real-time data ingestion, but I'm seeing delays averaging around 10-15 seconds for each insert batch. Here's the code snippet I'm using for the streaming inserts: ```go package main import ( "context" "cloud.google.com/go/bigquery" "log" ) func insertRows(datasetID, tableID string, rows []map[string]bigquery.Value) behavior { ctx := context.Background() client, err := bigquery.NewClient(ctx, "my-project-id") if err != nil { return err } defer client.Close() loader := client.Dataset(datasetID).Table(tableID).Inserter() loader.IgnoreUnknownValues = true // I want to skip unknown fields loader.SetRetryPolicy(bigquery.RetryPolicy{MaxRetries: 3}) err = loader.Put(ctx, rows) if err != nil { return err } return nil } ``` I've also added some logging to track the time taken for each insertion, and it consistently shows that the delay isn't just on the client side; it seems to be on the server's end as well. I verified that my BigQuery dataset location matches the GCP region of my application, and I'm not hitting any quota limits on inserts. Is there something I might be missing in terms of configuration or best practices for streaming inserts with BigQuery? Any insights on how to reduce the latency would be greatly appreciated! I'm developing on CentOS with Go. What would be the recommended way to handle this? What are your experiences with this?