How to Run a Python ETL Script in Azure Data Factory and Choose the Best Approach?
I'm currently working on an ETL process where I need to run a Python script within Azure Data Factory (ADF). The script involves data extraction, transformation, and loading (ETL) tasks. I’m aware that ADF supports various ways to execute custom code,…
Need help in understanding times in ADF's tumbling window trigger
Hi Team, I'm confused by many times in tumbling window trigger. We have at least 4 times for tumbling window trigger. @trigger().outputs.windowStartTime @trigger().outputs.windowEndTime @trigger().scheduledTime @trigger().startTime Besides, there…
In ADF, All Linked Services not showing up in Git Mode from Live Mode
There are total 4 linked services created in my ADF and its on live mode. I make a git configuration and set the ADF to new branch along with providing publish branch. But only 2 linked services showed up leaving behind a LS with setup on azure sql db…
ADF Copy Activity failed with ORA-00928: missing SELECT keyword
ADF Copy activity is failing with following error. Its happening after copying certain amount of data. So don't think its issue with pipeline. Also same pipeline works in another environment(PROD/DEV) so mostly its data issue. How to identify what is…
Can't able to connect the VM installed postgresql from ADF
Hi Team, We are unable to access the PostgreSQL from the ADF linked server. it's a azure PostgreSQL flexible server, and we have enabled the "Allow public services" setting within Azure for this server. However, we are still unable to create a…
Error when uploading CSV files to Azure blob storage using ManagedIdentity with C# and GraphQLAPI. "Total size of resourceFiles is using 147% of available space after processing."
Hello All, I have created a console app using C# where I try to fetch the data from GraphQLAPI and store the data in csv files. post that it would be uploaded to the Azure blob storage using ManagedIdentity. This complete code work fine in my local VS…
how to retrieve the path to workspace_publish branch OR any branch in devops , use that path in azure powershell script to read file from that branch.
I want to retrieve path to the branch and folder inside it, basically i wan to read the jsons here and change the dev keyword with uat and save the file back to the sam folder in workspace_publish branch. I can change dev to uat, but am not able to…
CSV to XML conversion in databricks which have some blank values as well in csv
I am converting CSV data to xml and that CSV data has some blank values as well for a few columns let's take an example there are 4 columns in CSV and out of that for a row(record) 1 colom value is blank , so as an output in xml, I am getting a missing…
can use a dataset for pipeline but can't use for dataflow in Azure Data Factory
Created a dataset for Azure SQL and it works fine in any pipelines of Azure Data Factory. However, it gives me the connection error from Data Flows - refer to the below. The Azure SQL is configured to be accessible from any Azure services. Spark…
Using CRM as a source in the ADF Metadata-driven copy task results in Fetch XML syntax errors
I am using the Metadata-driven copy task in Azure Data Factory (ADF). The process works as expected when using SQL server as a source, but when I use CRM as a source, I get source query syntax errors when the pipeline executes. The problem seems to be…
How can I get all files inside a drive irrespective of folder structure in ADF?
I want to copy files files from sharepoint drive which has lots of nested folders. Maximum hierarchy of folders is for 12 levels. Currently I'm using below endpoint in ADF's Web activity as it was mentioned in some of the articles that it provides every…
The Integration Runtime (Self-hosted) node has encountered an error during registration.
I am not able to register Self Hosted Integration Runtime on my Machine. This is the error I am getting : The Integration Runtime (Self-hosted) node has encountered an error during registration. Unable to determine the Integration Runtime (Self-hosted)…
How to copy the data from Amazon S3 to Azure SQL database incrementally.
Hello team, I wanted to wanted to copy data from Amazon S3(JSON format) to Azure SQL database. I wanted to learn the mechanism for incrementally updating the data from Amazon S3 to Azure SQL. Also, the environment has only SHIR deployed, so data flows…
Configuring Linked Service for Dev to Prod Movement in Azure Synapse Analytics
In Azure Synapse Analytics, I have a linked service of type Azure SQL Database Managed Instance, and I have two linked services, one for dev and one for prod. How can I move from dev to prod without changing the linked service manually? How can I make it…
Azure Data Factory - jobs not even running?
I have a persistent issue with ADF since Friday. Occasionally our pipelines will show "in progress" however the activities just wont move. Not even in a queued state, they just won't start at all. Right now we have a pipeline that has been…
Azure to AWS
Hello We need to transfer files from ADLS to AWS (S3 bucket) for a SAS application hosted in third party in batches. We need to ensure data security and best practices. My understanding, we can use ADF to create a linked service for AWS S3 but IT DOES…
How to use next link for pagination in adf (I have to use pagination for web activity)
Hi, I am trying to build up an adf pipeline to pull data from NetSuite. The result of the api call of NetSuite is like this in postman As we can see that this returns next link with the data and now we have to us this next link for pagination in adf. …
Authentication type missing from linked service in Azure Data Factory when
Hi, MS released a new connector for PostgresSQL in Linked Services of Azure Data Factory v2. We could create a new linked service in the development branch that worked ok. But when we deploy the linked service to the ADF (connected to Git), the…
Azure Data Factory Oracle to Parquet datatype mismatch
Hi Team, I have build a dynamic pipeline to load the data from Oracle tables to datalake as Parquet files. Now, when I am trying to copy the data from source to sink through CopyActivity, then I am not able to convert the decimal datatype of the…
How to add Azure SQL Database as linked service to Azure Data Factory
I keep running into errors adding Azure SQL as a linked service to Data Factory. I'm new to Azure so please bear with me. Following all the online guides, I've done all of the following: On the SQL Server side: Under Microsoft Entra ID, enabled…