👀 Views: 22 💬 Answers: 1 📅 Created: 2025-08-22

google-cloud-dataflow apache-beam gcs Java

I'm testing a new approach and I'm attempting to set up I tried several approaches but none seem to work... I'm currently working on a Dataflow job that processes files stored in Google Cloud Storage, but I'm encountering a frustrating error: `java.io.FileNotFoundException: No such file or directory`. My Dataflow pipeline is configured to read from a specific bucket, and I'm using the `TextIO` source to read the files. I've verified that the files exist in the bucket and that the specified path is correct, but the job fails during execution with the aforementioned error. Here's a simplified version of my pipeline code: ```java import org.apache.beam.sdk.Pipeline; import org.apache.beam.sdk.io.TextIO; import org.apache.beam.sdk.transforms.DoFn; import org.apache.beam.sdk.transforms.MapElements; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.TypeDescriptor; public class MyDataflowJob { public static void main(String[] args) { Pipeline p = Pipeline.create(); // Replace with your GCS path String inputFilePath = "gs://my-bucket/input/*.txt"; p.apply("ReadFiles", TextIO.read().from(inputFilePath)) .apply("ProcessData", ParDo.of(new DoFn<String, String>() { @ProcessElement public void processElement(ProcessContext c) { String line = c.element(); // Processing logic here c.output(line.toUpperCase()); } })) .apply("WriteResults", TextIO.write().to("gs://my-bucket/output/result")); p.run().waitUntilFinish(); } } ``` I also checked the permissions and can confirm that the Dataflow service account has the necessary read permissions for the input bucket. The file names are correct, and I've tested the wildcard pattern in the GCS console. Additionally, I’m using Apache Beam SDK version 2.34.0. When I run the job, it fails with the log message: ``` java.io.FileNotFoundException: No such file or directory: gs://my-bucket/input/*.txt ``` I've tried both specific files and wildcard patterns, but I keep hitting the same wall. Could there be a misconfiguration in my Dataflow setup, or is there anything about using wildcards with `TextIO` that I might be overlooking? Any insights would be greatly appreciated! Is there a better approach? The stack includes Java and several other technologies. Has anyone dealt with something similar? Am I missing something obvious?