Duplicate Activity in Purview

Hargopal Singh 20

Hi Team,

I created a pipeline in Synapse and the required lineage is visible in Purview i.e. flow of data from different activities. But When I edit the pipeline (rename column in Table or update general description) instead of updating the existing lineage, a new branch/activity is created. Attached is the snapshot from Purview lineage.

Is there a way to get this issue resolved?

Regards,

Hargopal

PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee

2024-05-07T08:22:04.97+00:00
@Hargopal Singh - Thanks for the question.

Could you please share the below details for investigating furthur:

Lineage screenshot before updating.

Lineage screenshot after updating.

Did you run the pipeline after editing the pipeline?

Looking forward for your response.
Hargopal Singh 20 Reputation points

2024-05-07T14:46:41.5466667+00:00

Hi Pradeep,

Below are the snapshot of Lineage before and after edit. Yes, I ran the pipeline after edit. Also, If I change the column name then there is no duplicate relationship or branch in lineage but If I update the general description of pipeline (e.g. Copy activity1 --> New Copy activity) then I can see new relationship/ branch.

Lineage after Edit.png

Lineage Before Edit.png
PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee

2024-05-08T06:38:13.1133333+00:00

@Hargopal Singh - Thank you for providing the snapshots. Based on the snapshots, it seems like the issue is related to how Purview is capturing the lineage for the updated pipeline.

When you update the general description of the pipeline, Purview is treating it as a new activity and generating a new lineage for it. This is because Purview captures lineage based on the activity name and the run ID. When you update the general description of the pipeline, the activity name changes, and Purview treats it as a new activity.

To avoid this issue, you can try the following:

Instead of updating the general description of the pipeline, try updating the activity name or description of the individual activities within the pipeline. This should not create a new activity in Purview and should update the existing lineage.

If you need to update the general description of the pipeline, you can try deleting the old activity in Purview and then rerunning the pipeline. This should generate a new activity in Purview with the updated general description and should not create a duplicate lineage.

I hope this helps! Let me know if you have any further questions.
Hargopal Singh 20 Reputation points

2024-05-08T14:23:04.3233333+00:00

Hi Pradeep,

Thanks for the details. I am not updating the general description of Pipeline. I am editing the Activity name / description.

Regards,

Hargopal
Hargopal Singh 20 Reputation points

2024-05-08T14:26:29.3033333+00:00

Hi Pradeep,

Even, I am not updating the Pipeline Description instead I am editing activity Name/description.
PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee

2024-05-20T05:51:43.1366667+00:00

@Hargopal Singh - Just checking in to see if the below answer helped. If this answers your query, do click Accept Answer and Yes for was this answer helpful. And, if you have any further query do let us know.

Accepted answer

PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee

2024-05-09T05:53:00.6166667+00:00
@Hargopal Singh - I apologize for the confusion. If you are editing the activity name or description, then Purview should update the existing lineage instead of creating a new branch.

However, if you are still seeing duplicate branches in the lineage, it could be due to a bug or issue with Purview. In this case, I would recommend to open a support ticket for further assistance.

You can also try the following troubleshooting steps:

Check if the activity name or description is being updated correctly in Synapse. If there is an issue with Synapse, it could be causing the duplicate branches in Purview.

Check if there are any errors or warnings in the pipeline run history in Synapse. If there are any issues with the pipeline run, it could be causing the duplicate branches in Purview.

Try deleting the duplicate branch in Purview manually. To do this, select the duplicate activity in Purview and click on the "Delete" button. This will remove the duplicate branch from the lineage.

I hope this helps! Let me know if you have any further questions.
Please sign in to rate this answer.
Hargopal Singh 20 Reputation points

2024-05-09T11:49:24.0266667+00:00

Thanks Pradeep for the details.

PRADEEPCHEEKATLA-MSFT 80,491 Reputation points Microsoft Employee

2024-05-20T05:52:41.5266667+00:00

@Hargopal Singh - Glad to know it helped. Please don’t forget to Accept Answer and Yes for "was this answer helpful" wherever the information provided helps you, this can be beneficial to other community members.
Sign in to comment

Share via

Duplicate Activity in Purview

0 additional answers