👀 Views: 1168 💬 Answers: 1 📅 Created: 2025-06-09

gcp google-cloud-storage aws-s3 data-transfer permissions Python

I'm sure I'm missing something obvious here, but I tried several approaches but none seem to work. I'm trying to set up a Google Cloud Storage Transfer Service to transfer files from an AWS S3 bucket to a Google Cloud Storage bucket. I've set up the transfer job using the GCP console and configured the necessary permissions for both AWS and GCP. When I run the transfer job, I encounter an 'InvalidArgument' behavior with the following message: ``` behavior: Invalid argument: The provided source is not valid. ``` I have verified that the S3 bucket path is correct and that I am using the appropriate IAM roles for the GCP service account. Here’s the code I used to create the transfer job: ```python from google.cloud import storagetransfer_v1 client = storagetransfer_v1.StorageTransferServiceClient() transfer_job = storagetransfer_v1.TransferJob( description="Transfer from S3 to GCS", project_id="my-gcp-project", transfer_spec=storagetransfer_v1.TransferSpec( aws_s3_data_source=storagetransfer_v1.AwsS3Data( bucket_name="my-s3-bucket", aws_access_key=storagetransfer_v1.AwsAccessKey( access_key_id="my-access-key-id", secret_access_key="my-secret-access-key" ) ), gcs_data_sink=storagetransfer_v1.GcsData( bucket_name="my-gcs-bucket" ) ), status=storagetransfer_v1.TransferJob.Status.ENABLED ) response = client.create_transfer_job(transfer_job) print(response) ``` I’ve double-checked the AWS credentials and the IAM permissions, and they seem correct. I also ensured that the S3 bucket is in the same region as the GCP storage bucket. Is there something I'm missing, or any specific configuration that I should be aware of that could lead to this behavior? Any insights would be greatly appreciated. I'm developing on Debian with Python. I'm developing on Debian with Python.