AWS Glue Job scenarios with 'NoSuchJob' scenarios When Triggered by EventBridge Rule
I recently switched to After trying multiple solutions online, I still can't figure this out. I'm working with an scenario where my AWS Glue job fails to trigger when called by an EventBridge rule, and I receive the behavior 'NoSuchJob'. I've double-checked the job name and the ARN, but everything seems correct. Here’s my setup: 1. **EventBridge Rule**: I created a rule that should trigger on an S3 Put event, and it's configured to target my Glue job. ```json { "Source": ["aws.s3"], "DetailType": ["AWS API Call via CloudTrail"], "Detail": {"eventSource": ["s3.amazonaws.com"], "eventName": ["PutObject"]} } ``` 2. **Glue Job Configuration**: My Glue job is defined as follows: ```python import sys from awsglue.transforms import * from awsglue.utils import getResolvedOptions from pyspark.context import SparkContext from awsglue.context import GlueContext from awsglue.job import Job args = getResolvedOptions(sys.argv, ['JOB_NAME']) glueContext = GlueContext(SparkContext.getOrCreate()) job = Job(glueContext) job.init(args['JOB_NAME'], args) # Your ETL logic here job.commit() ``` 3. **AWS CLI Command to Trigger Job**: When testing manually via the AWS CLI, I'm using: ```bash aws glue start-job-run --job-name my-glue-job ``` And this works fine, which leads me to believe the job itself is configured correctly. However, when EventBridge tries to trigger the job, I receive the following behavior in the logs: ``` An behavior occurred (NoSuchJob) when calling the StartJobRun operation: The job my-glue-job does not exist. ``` I've confirmed that the job is present in the Glue console and that I'm using the exact job name. I'm also aware that the IAM role associated with the EventBridge rule has the necessary permissions to invoke the Glue job. I've tried redeploying the EventBridge rule and the Glue job, but the scenario continues. Is there something subtle I might be missing here, or could it be an scenario with how the EventBridge rule is configured? Any suggestions would be greatly appreciated! This is part of a larger API I'm building. I'm working on a service that needs to handle this. Thanks in advance! I recently upgraded to Python 3.11.