Module 1: Introduction to Azure Data Factory
- Overview of Data Integration
- What is ETL/ELT?
- Data pipeline concepts
- ADF in the Azure ecosystem
- ADF Core Concepts
- Linked services
- Datasets
- Pipelines
- Activities
- Triggers
- Integration runtime
- ADF Use Cases
- Data migration
- Data warehousing
- Hybrid data integration
- Big data processing
Module 2: Getting Started with ADF
- Creating an ADF Instance
- Azure portal setup
- ARM templates
- PowerShell/CLI deployment
- ADF UI Tour
- Authoring canvas
- Monitoring hub
- Management hub
- Your First Pipeline
- Copy data wizard
- Manual pipeline creation
- Basic debugging
Module 3: Data Movement in ADF
- Copy Activity Deep Dive
- Supported data sources/sinks
- Mapping data flows
- Schema drift handling
- Performance Optimization
- Parallel copies
- Data Integration Units (DIUs)
- Partition options
- Hands-on Lab: Migrate data from Blob Storage to SQL Database
Module 4: Data Transformation
- Mapping Data Flows
- Visual data transformation
- Source/sink transformations
- Derived columns
- Aggregations
- Joins/lookups
- External Transformations
- Azure Databricks integration
- HDInsight activities
- Stored procedure activities
- Hands-on Lab: Transform sales data with data flows
Module 5: Advanced Pipeline Concepts
- Control Flow Activities
- If conditions
- ForEach loops
- Until activities
- Execute pipeline activities
- Error Handling
- Activity retries
- Failure paths
- Dependency conditions
- Parameters & Variables
- Pipeline parameters
- System variables
- Custom variables
Module 6: Scheduling & Orchestration
- Triggers
- Schedule triggers
- Tumbling window triggers
- Event-based triggers
- Monitoring & Alerting
- Pipeline runs monitoring
- Setting up alerts
- Log analytics integration
- Hands-on Lab: Schedule a monthly data warehouse refresh
Module 7: Security & Best Practices
- Security Features
- Managed identities
- Key Vault integration
- Network security
- Performance Tuning
- Partitioning strategies
- Caching options
- Monitoring performance
- CI/CD for ADF
- Git integration
- ARM template deployment
- DevOps pipelines
Module 8: Real-World Scenarios
- Hybrid Data Integration
- Self-hosted IR setup
- On-premises to cloud scenarios
- Big Data Patterns
- Processing large datasets
- Delta lake integration
- Final Project: End-to-end data pipeline from source to Power BI