GCP Dataflow job scenarios with 'how to to resolve hostname' when accessing external API from worker nodes
I'm testing a new approach and I've spent hours debugging this and I've looked through the documentation and I'm still confused about I'm running a Google Cloud Dataflow job to process data and make calls to an external API. However, I'm working with an scenario where my worker nodes are producing the behavior 'Unable to resolve hostname' for the API endpoint. I’ve ensured that the external API is reachable from my local environment, and I’m using a custom Docker image that includes all necessary dependencies. The Dataflow job is defined in Python using the Apache Beam SDK version 2.34.0, and my code snippet looks like this: ```python import apache_beam as beam def call_external_api(data): import requests response = requests.get('https://api.example.com/data') # External API return response.json() with beam.Pipeline() as pipeline: (pipeline | 'Create Input Data' >> beam.Create(['data1', 'data2']) | 'Call API' >> beam.Map(call_external_api)) ``` I've also set the Dataflow job to run in a VPC using a custom network and subnet that I verified has the necessary egress rules in place to allow outbound HTTP requests. I checked the environment variables and network configurations, and everything seems set up correctly. Despite these settings, the logs show the following behavior message from the worker nodes: ``` behavior: Unable to resolve hostname 'api.example.com' ``` I’ve tried switching to a different API endpoint and also increasing the timeouts for the requests, but the scenario continues. Does anyone have insights into what could be misconfigured in my Dataflow setup that would lead to this DNS resolution failure? Any pointers on debugging this would be greatly appreciated. This is part of a larger CLI tool I'm building. Any ideas what could be causing this? What are your experiences with this?