👀 Views: 33 💬 Answers: 1 📅 Created: 2025-06-07

gcp dataflow apache-beam Java

I'm working on a project and hit a roadblock. This might be a silly question, but I'm sure I'm missing something obvious here, but I'm working on a personal project and I have been trying to implement a Dataflow pipeline using Apache Beam (version 2.38.0) that processes a stream of data from Pub/Sub and applies a tumbling window operation..... However, I keep working with an behavior that says `java.lang.IllegalArgumentException: Invalid window` when executing the pipeline. This seems to occur when the pipeline attempts to finalize the windows for processing. My pipeline code is structured like this: ```java PipelineOptions options = PipelineOptionsFactory.create(); Pipeline p = Pipeline.create(options); p.apply("ReadFromPubSub", PubsubIO.readStrings().fromTopic("projects/my-project/topics/my-topic")) .apply("TumblingWindow", Window.<String>into(TimeWindows.of(Duration.standardMinutes(10)))) .apply("CountWords", Count.globally()) .apply("FormatResults", MapElements.via(new SimpleFunction<Long, String>() { @Override public String apply(Long count) { return "Count: " + count; } })) .apply("WriteToBigQuery", BigQueryIO.writeTableRows() .to("my-project:my_dataset.my_table") .withFormatFunction(new SerializableFunction<TableRow, TableRow>() { @Override public TableRow apply(TableRow row) { return row; } })); p.run().waitUntilFinish(); ``` I've checked the input data and confirmed that it arrives as expected. I also verified that I'm using the correct windowing strategy. However, I suspect the question may be related to the timestamps assigned to the incoming Pub/Sub messages. I'm using the default timestamp assigned by Pub/Sub, but I haven't explicitly set a timestamp in my code. To debug, I tried adding a `WithTimestamps` transform to assign the timestamps manually: ```java .apply("AssignTimestamps", WithTimestamps.of((String element) -> Instant.now())) ``` Unfortunately, this didn't resolve the scenario, and I still see the same behavior. I'm unsure how to handle the timestamps correctly or if there's something I might be missing in the windowing logic. Any suggestions on how to resolve this behavior would be greatly appreciated! My development environment is Windows. Any ideas what could be causing this? My development environment is Windows. Am I missing something obvious? This is part of a larger CLI tool I'm building. What am I doing wrong? How would you solve this? The stack includes Java and several other technologies. I appreciate any insights!