OCI Data Integration Task scenarios with 'Internal Server scenarios' on Large Dataset
I'm testing a new approach and I've spent hours debugging this and After trying multiple solutions online, I still can't figure this out... I'm sure I'm missing something obvious here, but I'm currently working on a data integration task using Oracle Cloud Infrastructure (OCI) Data Integration service, and I'm working with an 'Internal Server behavior' when trying to process a large dataset (around 10 million rows). The integration task is designed to copy data from an OCI Object Storage bucket to an Autonomous Database. I've set up the task with a mapping that specifies the source and target tables correctly. When I attempt to run the integration task, I receive the following behavior message: ``` behavior Code: 500 Internal Server behavior Message: Unexpected behavior occurred while executing the integration task. ``` To troubleshoot, I checked the following: - The OCI Object Storage bucket policies and confirmed that the Data Integration service has the necessary permissions to access the data. - I reviewed the task's logs, but they only indicate the internal server behavior without further details. - I also tried splitting the dataset into smaller chunks, but even a chunk of 1 million rows fails with the same behavior. Here’s a snippet of my integration mapping configuration: ```json { "source": { "type": "ObjectStorage", "bucket": "my-bucket", "file": "large-dataset.csv" }, "target": { "type": "AutonomousDatabase", "connection": { "username": "myusername", "password": "mypassword", "database": "mydb" }, "table": "target_table" } } ``` I've also verified that the Autonomous Database is running and can accept connections. Is there a limit on the data size that the OCI Data Integration service can handle in a single task? Or could there be other factors contributing to this internal server behavior? Any insights or advice would be greatly appreciated! For context: I'm using Json on Linux. Is there a better approach? For context: I'm using Json on Windows. Am I missing something obvious? This is happening in both development and production on Ubuntu 20.04. Thanks in advance! For reference, this is a production microservice.