👀 Views: 0 💬 Answers: 1 📅 Created: 2025-06-18

google-cloud-platform dataflow pubsub python

I'm maintaining legacy code that This might be a silly question, but I'm trying to run a Google Cloud Dataflow job that reads messages from a Pub/Sub subscription, but I'm working with an behavior that says 'Invalid Pub/Sub Subscription'. I have verified that the subscription exists and that my service account has the necessary permissions. Here's the snippet of my Dataflow job in Python: ```python import apache_beam as beam from apache_beam.options.pipeline_options import PipelineOptions, GoogleCloudOptions, SetupOptions project_id = 'my-gcp-project' subscription = 'projects/my-gcp-project/subscriptions/my-subscription' options = PipelineOptions() options.view_as(GoogleCloudOptions).project = project_id options.view_as(GoogleCloudOptions).streaming = True with beam.Pipeline(options=options) as p: (p | 'ReadFromPubSub' >> beam.io.ReadFromPubSub(subscription=subscription) | 'ProcessMessages' >> beam.Map(lambda msg: print(msg))) ``` When I run the job, I see the following behavior in the logs: ``` behavior:root:Invalid Pub/Sub Subscription: projects/my-gcp-project/subscriptions/my-subscription ``` I have checked that the Pub/Sub subscription is indeed active, and I can publish messages to it without issues. Additionally, my service account has been granted both 'pubsub.subscriber' and 'pubsub.viewer' roles. I've also tried using the subscription name without the project prefix, but that didn't help either. Has anyone faced this scenario or know what could be causing this behavior? Any insights would be appreciated! For context: I'm using Python on Ubuntu. Thanks in advance! This is part of a larger REST API I'm building. Thanks in advance!