Azure Data Factory Pipeline scenarios with 'Invalid Column Name' scenarios on Incremental Load
I'm optimizing some code but I've been struggling with this for a few days now and could really use some help... I'm trying to set up an incremental load in Azure Data Factory (ADF) from an Azure SQL Database to an Azure Blob Storage using a Copy Data activity. Everything was working well until I added a new column to the source table, which is causing the pipeline to unexpected result with the behavior message: `Invalid column name 'NewColumn'`. I've updated my source dataset to include this new column, but the pipeline still throws the behavior when it runs. Here's the relevant part of my ADF pipeline JSON definition: ```json { "name": "CopyFromSQLToBlob", "activities": [ { "name": "CopyActivity", "type": "Copy", "inputs": [ { "referenceName": "SourceDataset", "type": "DatasetReference" } ], "outputs": [ { "referenceName": "BlobOutputDataset", "type": "DatasetReference" } ], "source": { "type": "SqlSource", "query": "SELECT Id, Name, NewColumn FROM MyTable WHERE LastModified > @lastRunTime" }, "sink": { "type": "BlobSink" } } ] } ``` I verified that the new column `NewColumn` exists in the SQL table and is correctly spelled in the query. I also checked the linked service connection string for any issues. When I run the pipeline manually, it fails immediately with the behavior mentioned above. I tried using different query approaches, such as simply selecting all columns or using specific column names, but it hasn't resolved the scenario. Is there something specific in ADF that needs to be done when adding new columns to the source dataset, or am I missing a configuration step? I'm using Azure Data Factory V2 and the pipeline is set to trigger based on a tumbling window schedule. I'm working on a API that needs to handle this. I recently upgraded to Json 3.11. Thanks for your help in advance!