GCP BigQuery job timing out when processing large datasets with Java client library
I'm updating my dependencies and I'm getting frustrated with This might be a silly question, but This might be a silly question, but I'm currently experiencing timeouts when trying to run a BigQuery job using the Java client library for a dataset that's approximately 15GB. The query runs fine in the BigQuery console, but when I execute it through my application, it fails after about 10 minutes with the behavior message: `com.google.api.gax.rpc.DeadlineExceededException: Deadline exceeded while waiting for operation`. I've tried increasing the timeout settings in my Java client as follows: ```java BigQuery bigquery = BigQueryOptions.newBuilder().setProjectId("my-project-id").build().getService(); JobConfiguration queryConfig = JobConfiguration.newBuilder("SELECT * FROM `my_dataset.my_table` WHERE condition").build(); JobId jobId = JobId.of(UUID.randomUUID().toString()); JobInfo jobInfo = JobInfo.newBuilder(queryConfig).setJobId(jobId).build(); // Setting timeout to 20 minutes Job job = bigquery.create(jobInfo, BigQuery.JobOption.timeout(20 * 60 * 1000L)); ``` However, that doesn't seem to resolve the scenario. The same query runs without issues in the GCP console which makes me wonder if there's something different about how the client library handles the job execution. I've also checked the quota and limits for my project, and it appears that I'm well within the allowed ranges. Additionally, I've tried simplifying the query to see if the complexity might be causing the timeout, but even a basic `SELECT COUNT(*)` on the same dataset fails. Is there something I might be missing in the configuration or any best practices for handling large datasets with the BigQuery Java client? My development environment is Linux. What's the best practice here? I'm working with Java in a Docker container on Ubuntu 22.04. What's the correct way to implement this? I've been using Java for about a year now. I recently upgraded to Java 3.11. Am I approaching this the right way? I'm coming from a different tech stack and learning Java.