Azure Data Factory: Getting 'how to insert duplicate key' scenarios when copying data from SQL to Cosmos DB
I'm a bit lost with I'm getting frustrated with I'm currently working on an Azure Data Factory pipeline that copies data from an Azure SQL Database to Azure Cosmos DB. However, I keep running into the behavior message 'want to insert duplicate key' when executing the pipeline. The pipeline is set up to copy data based on a specific schedule, and I'm using the built-in copy activity. Here's the JSON definition of my pipeline: ```json { "name": "CopySqlToCosmosDB", "properties": { "activities": [ { "name": "CopyActivity1", "type": "Copy", "inputs": [ { "referenceName": "SqlInputDataset", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "CosmosDBOutputDataset", "type": "DatasetReference" } ], "source": { "type": "SqlSource" }, "sink": { "type": "CosmosDbSink", "writeBehavior": "merge" } } ] } } ``` In my Cosmos DB, I have a unique index set on one of the fields, which is necessary for my application. I've tried changing the `writeBehavior` from `merge` to `upsert`, but that didn't solve the scenario. I also ensured that the data being copied does not contain duplicates in the SQL table, but it seems that the pipeline still attempts to insert duplicates. To troubleshoot, I've enabled logging in Azure Data Factory, but the logs don't provide much detail aside from the same behavior message. I've also checked the data flow and validated that the source data is distinct before execution, yet this scenario continues. Has anyone encountered this question before, or does anyone have insights on how to avoid duplicate key errors when copying data to Cosmos DB? Any help would be greatly appreciated! My development environment is Linux. What am I doing wrong? This is part of a larger microservice I'm building. Is there a better approach?