implementing Azure Data Factory Copy Activity Timeout When Loading Large Datasets
I'm maintaining legacy code that I'm currently working with a timeout scenario while using Azure Data Factory's Copy Activity to transfer large datasets from Azure SQL Database to Azure Blob Storage. The source database has around 10 million rows, and the current setup times out after 30 minutes, leading to incomplete data transfers. I've tried increasing the timeout settings in the Copy Activity configuration, but it doesn’t seem to have any effect. Here’s how I’ve configured the Copy Activity: ```json { "name": "CopyFromSqlToBlob", "type": "Copy", "typeProperties": { "source": { "type": "SqlSource", "sqlReaderQuery": "SELECT * FROM LargeTable", "timeout": "01:00:00" }, "sink": { "type": "BlobSink" }, "enableStaging": false, "timeout": "01:00:00" } } ``` Initially, I thought the scenario might be with the source query, so I ran the SELECT statement directly in SQL Server Management Studio, and it executed successfully without any timeouts. I also checked the Azure Data Factory pipeline logs and found the following behavior message: ``` The operation has timed out. ``` Additionally, I’ve verified that the integration runtime is set to auto-resume, and I’m using a self-hosted integration runtime for better performance. I'm aware that this could be related to throttling issues or the way the dataset is being processed, but I’m unsure how to optimize the pipeline further. Has anyone else experienced similar issues with Azure Data Factory and large data copies? Any advice on configuration changes or design patterns that could help avoid timeouts would be greatly appreciated. I'm developing on Ubuntu 22.04 with Json. Could someone point me to the right documentation?