AWS Glue Job scenarios with 'NoSuchKey' scenarios When Accessing S3 Bucket with IAM Role
Can someone help me understand I'm collaborating on a project where I'm currently working with an scenario with my AWS Glue job where it fails with a `NoSuchKey` behavior while trying to read data from an S3 bucket..... The specific behavior message I'm getting is: ``` behavior: An behavior occurred while calling "get_partition" with args: ["my_database", "my_table", {"partitionValues": ["2023-10-01"]}] - NoSuchKey: The specified key does not exist. ``` I've verified that the S3 bucket policy allows the necessary read permissions for the IAM role associated with the Glue job. Here’s the relevant policy attached to the IAM role: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::my-bucket/*", "arn:aws:s3:::my-bucket" ] } ] } ``` The Glue job is configured to run with Spark, and I am using Python 3.7 for my ETL scripts. The S3 paths are dynamically generated based on input parameters passed to the job, and I have confirmed that files do exist at the expected paths in the S3 bucket. Here's a snippet of how I'm constructing my S3 path: ```python import boto3 import datetime s3_bucket = 'my-bucket' partition_date = datetime.datetime.now().strftime('%Y-%m-%d') input_path = f's3://{s3_bucket}/data/{partition_date}/' ``` When I log the `input_path`, it outputs the correct S3 URL. However, the job still need to find the key. I've double-checked the S3 console, and the file does exist at `s3://my-bucket/data/2023-10-01/my_file.csv`. I've also looked into AWS Glue's job logs, but there are no additional clues there. Is there something specific I might be missing in terms of Glue configurations or IAM role permissions, or could it be an scenario with the way I'm specifying the S3 key in my Glue job? My development environment is Windows. I'm on CentOS using the latest version of Python. I'd love to hear your thoughts on this. I recently upgraded to Python 3.11.