GCP Dataflow Job scenarios with 'No input sources' scenarios Despite Valid Pipeline Configuration
I'm stuck trying to I'm deploying to production and I'm prototyping a solution and I'm running a Dataflow job using Apache Beam (version 2.35.0) and I'm working with a frustrating scenario where the job fails with the behavior message 'No input sources' even though I have defined a valid source in my pipeline... My pipeline reads from a Pub/Sub topic and I'm using a subscription that I know has messages available. Here's a snippet of my pipeline code: ```python import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions class MyPipeline: def run(self): options = PipelineOptions( runner='DataflowRunner', project='my-gcp-project', region='us-central1', temp_location='gs://my-bucket/temp', staging_location='gs://my-bucket/staging', ) with beam.Pipeline(options=options) as p: (p | 'Read from PubSub' >> beam.io.ReadFromPubSub(subscription='projects/my-gcp-project/subscriptions/my-subscription') | 'Process Messages' >> beam.Map(lambda msg: print(msg)) ) if __name__ == '__main__': MyPipeline().run() ``` I've confirmed that the subscription is active and that there are messages in it. I also verified that my service account has the necessary permissions to access Pub/Sub. However, when I run the job, it starts but quickly fails with the aforementioned behavior. I tried increasing the worker machine type and changing the autoscaling settings, but that didn't help. I've also checked the Dataflow console, and it shows that the job was created but doesn't provide any additional insights into why it's failing. Is there something I might be missing in my configuration or setup? Any insights would be greatly appreciated! The project is a desktop app built with Python. Any ideas how to fix this? My development environment is macOS. For reference, this is a production desktop app.