Azure Data Factory Training This Week
Discount Code for Introductory + Advanced Courses
Use your subscriber coupon code - eodsubstack - at checkout and save 10%!
A Day of Azure Data Factory
Do you want to learn more about cloud data integration using Azure Data Factory?
Is your enterprise planning to migrate its data, databases, data warehouses (or some of them) to the cloud?
Do you currently use SSIS?
If you answered yes, A Day of Azure Data Factory is for you!
This class is delivered live and online 9:00 AM-4:30 PM ET 23 Apr 2025.
In this session we provision an Azure Data Factory (ADF) and demonstrate many uses for ADF in the modern data enterprise, including:
– Editing ADF pipeline JSON
– Version control integration
– Copying data with the copy data activity
– Parameters and use cases for parameterization (dynamic property values and expressions)
– Design patterns for the hybrid data estate
– Loading data from an on-premises data source
– Survey newer activities
– Debugging and troubleshooting ADF pipelines
– Looping, iteration, and conditionals
– Monitoring
– Execution automation (triggers + )
– ADF REST API
– Security required to interact with Azure Blob Storage, Azure SQL DB, REST API, integration runtimes, git, and on-premises (plus Key Vault)
– Provisioning and using integration runtimes
Advanced Azure Data Factory (ADF)
Course Outline*
*This is the first delivery of a new course, hence the low price! The outline and deliverables are subject to change.
Module 1: Introduction to Team Development in ADF (30 min)
Understanding ADF as a collaborative tool
Best practices for multi-developer environments
Git integration in ADF
Branching strategies (Git Flow, feature branching, hotfixes, etc.)
🛠 Hands-on: Setting up ADF source control in Azure DevOps or GitHub
Module 2: ADF Version Control and CI/CD Pipelines (1 hour)
Differences between ADF Live Mode vs. Git Mode
Managing deployments using ARM templates
Setting up a CI/CD pipeline in Azure DevOps
YAML Pipeline for ADF
🛠 Hands-on: Building and deploying an end-to-end CI/CD pipeline for ADF
Module 3: Advanced Triggers in ADF (45 min)
Understanding ADF triggers: Scheduled, Tumbling Window, and Event-Based Triggers
Tumbling Window Triggers: Concepts, Use Cases, and Late Arrivals
Event-Based Triggers: Monitoring file arrival in ADLS
🛠 Hands-on: Setting up a Tumbling Window Trigger
🛠 Hands-on: Setting up a Event-Based Trigger
Module 4: Notebook Execution in ADF (Fabric, Databricks, and Synapse) (45 min)
When to use Fabric, Databricks, or Synapse Notebooks
Executing Databricks Notebooks from ADF (Managed vs. External)
Notebook execution in Microsoft Fabric Pipelines
Connecting to Synapse Spark Notebooks
Passing parameters to notebooks from ADF
🛠 Hands-on: Triggering a Databricks/Fabric Notebook from ADF
Module 5: Debugging, ADF Expressions, and Monitoring (45 min)
Common ADF errors and troubleshooting techniques
ADF Expressions & Functions:
String functions (concat, substring, replace)
Date functions (formatDateTime, addDays, utcNow)
Collection functions (filter, union, intersection)
Debugging pipeline failures and JSON configuration issues
Using Azure Monitor, Log Analytics, and Application Insights
🛠 Hands-on: Writing advanced ADF expressions and debugging a failing pipeline
Module 6: Incremental Load Patterns in ADF (45 min)
Incremental vs. Full Load in ADF
Using Watermark Columns for Incremental Loads
Change Data Capture (CDC) in ADF
Implementing Delta Lake Merge in Fabric/Synapse
🛠 Hands-on: Building an Incremental Load Pipeline with CDC
Module 7: Advanced Orchestration Patterns in ADF (45 min)
Metadata-driven pipeline design
Pipeline chaining and dependency management
Looping & dynamic pipeline execution (ForEach, Until loops)
Dynamic parameterization for reusability
Hybrid Orchestration (Combining ADF with Logic Apps or Functions)
🛠 Hands-on: Implementing a metadata-driven dynamic pipeline
🛠 Hands-on: Start / stop an SSIS run-time via azure automation
Q&A and Wrap-Up (30 min)
Review key takeaways
Open discussion on real-world ADF challenges
Additional resources for further learning
Prerequisites
Experience with Azure Data Factory pipeline development
Basic understanding of JSON-based configurations in ADF
Familiarity with Azure DevOps, Databricks, and Synapse