OCI Data Integration: implementing Incremental Loading from Oracle DB to Autonomous Database
I'm currently working on a data integration pipeline using OCI Data Integration service to perform incremental loading from an on-premises Oracle Database (version 19c) to an Autonomous Database (ADB) instance on OCI. I've set up the connections correctly, but I'm working with issues with the incremental loading process. My configuration includes a staging table that tracks the last processed record's timestamp. However, when I run the integration task, I receive the following behavior: `behavior: Inconsistent data encountered: expected timestamp format but found NULL.` Here's a snippet of my integration mapping: ```json { "source": "OracleDB", "target": "AutonomousDB", "incrementalLoad": { "lastProcessedTimestamp": "2023-10-01T00:00:00Z" }, "filter": "timestamp >= :lastProcessedTimestamp" } ``` I've checked the data types and ensured that both source and target tables have compatible timestamp fields. I also validated the source data manually and confirmed there are no NULL timestamps before the specified last processed timestamp. Additionally, I've attempted to use the built-in data quality rules in OCI, but they do not seem to address this specific scenario. The integration task is set to run hourly, and the configuration appears correct, yet it fails consistently with the same behavior message. Anyone faced a similar scenario or have suggestions on how to troubleshoot this? Is there a specific configuration I might have missed, or is there a better approach for handling incremental loads in OCI Data Integration? This is part of a larger application I'm building. What's the best practice here? I've been using Json for about a year now.