Azure Data Factory: How to Configure a Pipeline to Handle Dynamic File Names in Blob Storage?
I've spent hours debugging this and I'm working on a project and hit a roadblock. I'm trying to create a pipeline in Azure Data Factory that can process dynamic file names stored in Azure Blob Storage. The scenario I'm working with is that the files I need to process are uploaded daily with names that follow a pattern, but also include a timestamp that changes each day (e.g., `data_2023-10-01.csv`, `data_2023-10-02.csv`). I've set up a Copy Activity to move data from Blob to a SQL Database, but I'm unsure how to dynamically reference the latest file. I've tried using a Lookup activity to get the list of files, but I keep running into issues with accessing the `childItems` property of the Blob Storage output. Here’s what I have so far in my pipeline JSON: ```json { "activities": [ { "name": "Get File List", "type": "Lookup", "typeProperties": { "source": { "type": "BlobSource" }, "dataset": { "referenceName": "MyBlobDataset", "type": "DatasetReference" } } }, { "name": "Copy Data", "type": "Copy", "dependsOn": [ { "activity": "Get File List", "dependencyConditions": ["Succeeded"] } ], "typeProperties": { "source": { "type": "BlobSource" }, "sink": { "type": "SqlSink", "dataset": { "referenceName": "MySqlDataset", "type": "DatasetReference" } } } } ] } ``` I expect to gather the latest file name from the Lookup activity and then use it in the Copy activity, but I receive an behavior: `want to read property 'childItems' of undefined`. I've also tried adding a `set variable` activity to store the latest filename, but I'm not sure how to reference that variable in the Copy activity. Could someone guide me on how to effectively fetch the latest file name and pass it dynamically to the Copy activity in Azure Data Factory? Any examples or insights would be greatly appreciated! This is part of a larger service I'm building. How would you solve this? This is part of a larger API I'm building. Any ideas what could be causing this? I'm using Json stable in this project.