Azure data factory interview questions

1. State the types of Integration Runtimes in ADF

Types of integration runtimes

Azure Integration Runtime: For cloud data movement and transformation.
Self-hosted Integration Runtime: For on-premises or hybrid data movement.
Azure-SSIS Integration Runtime: For running SSIS packages in the cloud.

2. How is Azure Data Factory different from SSIS?

ADF is cloud-based and supports modern cloud services while SSIS is primarily on-premises and focused on traditional ETL. ADF integrates seamlessly with other Azure services when compared to SSIS.

3. What are the different types of activities in ADF?

Types of activities in ADF

Data movement activities: Copy activity.
Data transformation activities: Mapping Data Flow, Wrangling Data Flow.
Control activities: ForLoop, If Condition, and Wait activities.
Custom activities: Executing custom code.

4. What is a pipeline in Azure Data Factory?

A pipeline is a logical collection of activities in ADF to perform data related tasks. Pipelines allow grouping of activities that defines the steps to process data and enable execution as a single unit.

5. What is the Copy Activity in Azure Data Factory?

In the Azure data factory, the copy Activity is used to copy data between different data stores, such as:

Azure Blob Storage,
SQL databases,
Other cloud,
On-premises sources.

6. What is the Mapping Data Flow in ADF?

In ADF, mapping Data Flow is a visual data transformation feature that allows users to design data transformations without writing code.

7. How can you monitor Azure Data Factory pipelines?

To monitor Azure data factory pipeline, use the ADF Monitor tab in the Azure portal. View pipeline run histories, activity runs, and debug logs. To get deeper insights, enable diagnostics logs in Azure Monitor.

8. What is a linked service in Azure Data Factory?

A linked service is like a connection string, which defines the connection information for a data source or destination to external resources or services. It is used to connect to data sources, compute services, and achieve reusability, authentication.

9. How do you handle errors in ADF pipelines?

To handle errors in ADF pipelines,

Use retry policies and implement Error Handling Activities like “If Condition” or “Set Variable.”
Leverage Activity dependency conditions like On Success, On Failure, and On Completion.

10. What is the role of triggers in Azure Data Factory?

Triggers automate the pipeline execution based on schedules, events, or tumbling window patterns.

11. What is the difference between a dataset and a linked service?

Dataset represents data structures (e.g., a table in SQL or a blob file). Linked Service connects configuration to data sources.

12. How is incremental loading implemented in Azure Data Factory?

Incremental loading is achieved using watermark columns or Delta detection techniques in pipelines.

13. Can you integrate Azure Data Factory with DevOps? How?

Yes, ADF integrates with Azure DevOps through:

Version control using Git.
CI/CD pipelines for deployment.

14. What are Integration Runtimes in Azure Data Factory?

Integration Runtimes (IR) are compute infrastructures used for data movement, transformation, and activity execution. The types of integration runtimes are:

Azure IR – Best for cloud-based data movement and transformation.

Self-Hosted IR – Used for on-premises data sources or private networks where direct cloud access is restricted.

15. What are the uses of self-hosted Integration Runtime?

Self-hosted Integration Runtime is used for on-premises data movement and connecting to private network data sources

16. How can you parameterize pipelines in ADF?

Parameterize pipelines by using parameters at pipeline and dataset levels, which can be passed during runtime to make the pipeline dynamic.

17. How do you secure data in Azure Data Factory?

Answer:

Use Managed Identity for authentication.
Enable TLS encryption for data in transit.
Use Azure Key Vault to manage secrets.

18. What are tumbling window triggers?

Tumbling window triggers execute pipelines in a fixed, periodic schedule (e.g., hourly, daily) and handle overlapping data windows gracefully.

19. How can you debug pipelines in Azure Data Factory?

Use Debug Runs to test pipelines interactively.
Monitor activity-level logs for insights.
Enable logging in Azure Monitor.

20. What is Azure Data Factory’s role in the modern data warehouse architecture?

ADF is responsible for orchestrating data ingestion, transformation, and movement from various sources into a centralized data warehouse like Azure Synapse Analytics for analytics and reporting.

21. How does ADF handle data movement?

ADF uses the Copy Activity to move data between supported sources and destinations, such as Azure Blob Storage, SQL databases, and on-premises systems.

22. What is the difference between Azure Data Factory and SSIS?

Azure Data Factory is a cloud-based service for orchestrating data movement and transformation.

SSIS (SQL Server Integration Services) is an on-premises ETL tool for data integration tasks. ADF can run SSIS packages using Azure-SSIS IR.

23. How do you handle errors and retries in ADF pipelines?

ADF allows configuring:

Retry Policies to specify retry attempts.
Error Handling using conditional logic, such as executing specific activities when errors occur.
Alerts and Logging for failures.

24. What is the difference between a scheduled trigger and a tumbling window trigger?

Scheduled Trigger: Runs at specified times, independent of previous runs.

Tumbling Window Trigger: Runs at specified intervals, but each window waits for the previous one to complete.

25. How can you monitor Azure Data Factory pipelines?

Monitor pipelines through the ADF Monitor tab, where you can view execution logs, pipeline runs, activity runs, and error details. Integration with Azure Monitor and Log Analytics is also available.

26. How do you secure data in Azure Data Factory?

Use Managed Identities for authentication.
Enable Azure Key Vault to store and retrieve secrets securely.
Implement Role-Based Access Control (RBAC).
Use Azure Private Link for private network connectivity.

27. How do you handle errors and retries in ADF pipelines?

ADF allows configuring:

Retry Policies to specify retry attempts.
Error Handling using conditional logic, such as executing specific activities when errors occur.
Alerts and Logging for failures.

28. What is the difference between Mapping Data Flow and Wrangling Data Flow?

Mapping Data Flow: Visual ETL tool for data transformation, optimized for large-scale data movement.

Wrangling Data Flow: Based on Power Query, designed for interactive data exploration and preparation.

29. How do you deploy Azure Data Factory pipelines across different environments?

Deploy Azure data factory pipelines using:

Azure DevOps CI/CD pipelines for automated deployments.
ARM Templates (Azure Resource Manager) to define and deploy ADF resources as code.

30. Can Azure Data Factory perform data transformation?

Yes, ADF supports data transformation using:

Data Flows (mapping data flows) for visually designing transformations.
Activities such as Azure Databricks, Azure SQL, and stored procedures for custom transformations.

31. How can you parameterize a pipeline in ADF?

use parameters at the pipeline level to pass dynamic values. These parameters can be assigned to datasets, Linked Services, or activities within the pipeline.

32. How do you schedule a pipeline to run every day at a specific time?

Create a Scheduled Trigger in Azure Data Factory and set it to run daily at a specified time. The trigger can be associated with one or more pipelines.

33. What is a Control Flow in Azure Data Factory?

Control Flow defines the execution sequence and logic within a pipeline. It orchestrates the activities, manages dependencies, and includes elements like conditional logic, loops (ForEach), and error handling.

34. How do you pass parameters to a pipeline run in ADF?

Pass parameters during a pipeline run by defining pipeline parameters and providing values when you trigger the pipeline manually or via triggers.

35. How does ADF handle incremental data loads?

ADF supports incremental data loads by:

Using Watermark Columns to track the latest data.
Utilizing Tumbling Window Triggers for periodic loads.
Implementing Change Data Capture (CDC)

36. What is the purpose of the Wait Activity in ADF?

The Wait Activity pauses pipeline execution for a specified duration. It can be useful for creating delays between tasks or waiting for external processes to complete.

37. What is the difference between Lookup Activity and Get Metadata Activity?

Lookup Activity retrieves data from a source (e.g., a table or file) and returns the results as output.
Get Metadata Activity retrieves metadata (e.g., file size, last modified date) from a data source.

38. How do you copy data from on-premises to Azure using ADF?

To copy on-premises data to Azure, use the Self-Hosted Integration Runtime (SHIR) to securely connect to on-premises data sources and the Copy Activity to transfer the data to Azure storage or databases.

39. How can you debug pipelines in Azure Data Factory?

ADF offers a Debug mode that allows you to run pipelines in real-time and view detailed logs and outputs for each activity to troubleshoot issues.

40. How do you manage version control in Azure Data Factory?

ADF supports version control integration with Azure Repos (Git) or GitHub. You can branch, merge, and track changes, and publish pipelines from your repository to the live environment.

41. What is the purpose of the Execute Pipeline Activity?

The Execute Pipeline Activity allows you to invoke another pipeline within a parent pipeline. This helps modularize workflows and reuse pipeline logic.

42. What are the advantages of using Azure Data Factory over traditional ETL tools?

Advantages include:

Scalability: Cloud-based and can handle large-scale data.
Cost-Effectiveness: Pay-as-you-go pricing model.
Integration: Supports diverse data sources (cloud and on-premises).
Orchestration: Easy to create complex workflows.
Serverless: No infrastructure to manage.

43. How do you manage version control in Azure Data Factory?

ADF supports version control integration with Azure Repos (Git) or GitHub. You can branch, merge, and track changes, and publish pipelines from your repository to the live environment.

44. How do you delete files from Azure Blob Storage using ADF?

use the Delete Activity to remove files from Azure Blob Storage. The activity can be configured with the target dataset and file path.

45. How can you handle dynamic file names in ADF?

Use pipeline parameters, expressions, and system variables like @utcnow() to generate dynamic file names in activities like Copy or Data Flow.

46. What is an activity dependency in ADF?

An activity dependency defines the order of execution between activities. You can set dependencies with conditions like Success, Failure, Completion, or Skipped.

47. How do you implement conditional logic in ADF pipelines

use the If Condition Activity to execute activities based on a condition. You can also use expressions to evaluate conditions dynamically.

48. What is a Data Flow Debug Mode in ADF?

Data Flow Debug Mode allows you to test and debug your data flows interactively with sample data, making it easier to troubleshoot and validate transformations.

49. How does ADF integrate with Azure Key Vault?

ADF integrates with Azure Key Vault to securely store and retrieve credentials, secrets, and connection strings, enhancing security and simplifying secret management.

50. How do you handle data schema changes in ADF?

ADF can handle schema changes by enabling the “Auto Mapping” option in the Copy Activity or using dynamic expressions in Data Flows to adapt to schema changes.

Request more information

Azure data factory interview questions