How to add Azure SQL Database as linked service to Azure Data Factory
I keep running into errors adding Azure SQL as a linked service to Data Factory. I'm new to Azure so please bear with me. Following all the online guides, I've done all of the following: On the SQL Server side: Under Microsoft Entra ID, enabled…
Delete records from a salesforce object using Azure data factory
Hi there, I am working on a ADF pipeline that transforms data from salesforce and finally i need to delete some records from salesforce object. adf does not have a built in feature to delete and i was wondering if we can utilize lookup activity to run …
How to use next link for pagination in adf (I have to use pagination for web activity)
Hi, I am trying to build up an adf pipeline to pull data from NetSuite. The result of the api call of NetSuite is like this in postman As we can see that this returns next link with the data and now we have to us this next link for pagination in adf. …
Azure ADF doesn't recognize changes in pipeline activity
Hi, I'm facing an issue with ADF when trying to change my pipelines. It doesn't accept my changes when it executes using the activity runtime, it only applies the changes in debug mode. Before the changes, my pipeline looks like this with the third Data…
InsertDate and UpdateDate column addition in Sink-Delta from dataflow.
I am using dataflow to Upsert the data to my delta lake. My requirement is to have two columns - 1. insertdate 2.updatedate . If there is a new record then both insertdate and updatedate column should get updated with current_date() values. And, if…
How can we connect Blackboard DDA (a SaaS using PostgreSQL to store data ) via Azure Synapse analytics using SSL certificate for authentication in SHIR for Private Vnet enabled networks.
Blackboard DDA is a SaaS application which uses PostgreSQL as DB to store data. The DB uses a SSL certificate key to authenticate and we have private Vnet enabled. How to create a linked service for this DB and where do we store certificate in SHIR. Are…
COPY ACTIVITY FROM SOURCE LIKED REST TO DESTINATION LOINED REST SERVICE
Want to copy from source table to destination table in the same log analytics workspace. Using Azure Data Factory. My configuration: Source Linked REST Service to read data from Log Analytics workspace w1 table t1 Destination Linked REST service to…
How to migrate to .Net 8?
How to increase the timeout of the Azure data factory pipleine
Getting the Integration run time error while created the linked service for Sql server in datafactory
Getting the Integration run time error while created the linked service for Sql server in data factory, can anyone help to solve this error Cannot connect to SQL Database. Please contact SQL server team for further support. Server:…
Why is the public ip range download blocked?
Hey, I use a script to download the public ip ranges from azure (https://www.microsoft.com/en-us/download/details.aspx?id=56519). If I view that URL in my browser, it displays and I am able to download the JSON. However, if I run my script that would…
How to store an array in a json file in ADF
I have a web activity with url drive/drive-id/items/folder-id/children then I have a filter activity that filters all folders and then I'm appending it in an array. Now what I want is to store all folder ids in a json file. and later I will load that…
Using CRM as a source in the ADF Metadata-driven copy task results in Fetch XML syntax errors
I am using the Metadata-driven copy task in Azure Data Factory (ADF). The process works as expected when using SQL server as a source, but when I use CRM as a source, I get source query syntax errors when the pipeline executes. The problem seems to be…
Data Factory Copy task getting stuck on a specific record count when copying
Hi All, I am copying data from a SAP Hana table on Prem to parquet file in ADLS, the table is only 30 000 000 big but when the copy task get to 12 000 000 odd records it seems to hang or stops copying. I have checked the source data and it is fine no…
Data Upsert Process in Azure SQL
I'm working on transferring a whole table from one database to several others, all in Azure SQL. To do this, I'm using a Lookup activity to get the list of server and database names. Then, I'm running a loop (ForEach) where I have a Copy Data activity.…
How to resolve invalid property name error in ADF copy activity?
ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'BadRequest'. Account: 'rmsdatalakestracc1mqa'. FileSystem:…
Custom libraries (wheel) for ADF Databricks Python activity run on serverless compute
I want to be able to execute Python scripts (via Databricks Python) from Azure Data Factory using serverless compute. Serverless compute does not support cluster level (compute scoped) libraries. In databricks workflows, it is being done as…
can use a dataset for pipeline but can't use for dataflow in Azure Data Factory
Created a dataset for Azure SQL and it works fine in any pipelines of Azure Data Factory. However, it gives me the connection error from Data Flows - refer to the below. The Azure SQL is configured to be accessible from any Azure services. Spark…
Recommended technologies for creating a dashboard of ongoing jobs.
I'm looking for recommendations on technologies to use for the following project. I need to create a dashboard of summarized job data from a network of hundreds of users. Our current application is used in the field for inputting and tracking job costs…
check if the value is a number and output in iif logic of derived transformation
I am checking if the input value is a number then output as '100' else 'invalid' in derived transformation in dataflow. In the documentation, I found 'isLong' function. But, how do we use this function in an iif…
Azure Data Flow Sink - Connection string is not valid or missing Server/User/Database properties
Could you please assist here to resolve this issue below ? We are trying to read Delta table and store it as table in Azure SQL Database as using ADF Data Flow. Source Delta table via inline dataset from ADLS Gen2 its working fine and slink is getting…