Azure Data Factory - Pipeline scenarios with 'Invalid Column Name' scenarios When Using Dataflow
I recently switched to Can someone help me understand I'm having trouble with I'm having trouble with I'm sure I'm missing something obvious here, but I've searched everywhere and can't find a clear answer....... I'm stuck on something that should probably be simple. I'm working with an scenario with Azure Data Factory where my data pipeline fails with an 'Invalid Column Name' behavior when trying to execute a dataflow transformation. I'm using a copy activity to pull data from an Azure SQL Database and then applying a dataflow to transform it before writing it back to another Azure SQL Database. The specific behavior message is: ``` Invalid Column Name: 'myColumn' ``` I've double-checked my source and destination schemas, and both databases have the column `myColumn` defined correctly. The dataflow is set up to aggregate and filter the data, but when I run the pipeline, it seems to unexpected result at the transformation stage. In my dataflow, I have a derived column transformation that looks like this: ```json { "name": "Derived Column", "type": "DerivedColumn", "columns": [ { "name": "myColumn", "expression": "toString(myColumn)"} ] } ``` I have verified that the column is coming from the source dataset correctly. I've also tried running the dataflow independently in the data factory debug mode, and it works without behavior. The only time it fails is when invoked through the pipeline. I've attempted to revalidate the linked services to ensure they are configured correctly and have also checked the integration runtime settings. Could this behavior stem from a mismatch in the data types between the source and destination databases, or should I be looking at something else? Any suggestions on how to troubleshoot this further would be appreciated! Am I missing something obvious? I'm working on a web app that needs to handle this. I'd really appreciate any guidance on this. For context: I'm using Json on macOS. Is there a better approach? Any ideas how to fix this? Am I missing something obvious? My development environment is CentOS. Am I missing something obvious? Am I approaching this the right way?