Visão Geral

Este curso foi desenvolvido para preparar profissionais para o exame oficial de certificação Databricks Certified Data Engineer Associate, abordando os conceitos, práticas e laboratórios necessários para atuar com engenharia de dados na plataforma Databricks. O treinamento contempla desde os fundamentos da arquitetura Lakehouse até a implementação de pipelines de dados utilizando Delta Lake, Spark SQL, DataFrames e workflows automatizados.

Ao longo do curso, os participantes desenvolverão competências práticas alinhadas aos objetivos oficiais do exame, realizando exercícios e simulados para maximizar as chances de aprovação na certificação.

Objetivo

Após realizar este curso Databricks Certified Associate Data Engineer Exam Preparation, você será capaz de:

• Compreender a arquitetura Lakehouse da Databricks
• Trabalhar com notebooks Databricks
• Desenvolver pipelines de engenharia de dados
• Manipular dados utilizando Spark SQL e DataFrames
• Implementar soluções utilizando Delta Lake
• Realizar ingestão de dados batch e streaming
• Configurar e monitorar workflows
• Gerenciar tabelas e metadados no Unity Catalog
• Aplicar boas práticas de performance e otimização
• Preparar-se para o exame oficial Databricks Certified Data Engineer Associate

Publico Alvo

• Data Engineers
• Data Architects
• Data Analysts
• BI Developers
• ETL Developers
• Cloud Engineers
• Profissionais que desejam obter a certificação Databricks Certified Data Engineer Associate

Pre-Requisitos

• Conhecimentos básicos de SQL
• Conceitos de bancos de dados relacionais
• Noções de Data Warehousing
• Conhecimentos básicos de Python
• Familiaridade com ambientes de nuvem (AWS, Azure ou GCP)

Materiais

Conteúdo Programatico

Module 1: Databricks Lakehouse Fundamentals

1. Introduction to Databricks Platform
2. Lakehouse Architecture
3. Data Engineering Concepts
4. Databricks Workspace Overview
5. Clusters and Compute Resources
6. Databricks Runtime
7. Workspace Administration Basics

Module 2: Working with Databricks Notebooks

1. Notebook Fundamentals
2. Multi-Language Support
3. Python in Databricks
4. SQL in Databricks
5. Notebook Workflows
6. Magic Commands
7. Parameterization Techniques

Module 3: Spark DataFrame Fundamentals

1. DataFrame Architecture
2. Reading Data Sources
3. Transforming Data
4. Filtering and Aggregation
5. Joining DataFrames
6. Working with Nested Data
7. DataFrame Best Practices

Module 4: Spark SQL Fundamentals

1. SQL Warehouses
2. Query Execution
3. SQL Functions
4. Views and Temporary Views
5. Advanced Queries
6. Data Manipulation Operations
7. Query Optimization Basics

Module 5: Delta Lake Essentials

1. Introduction to Delta Lake
2. Delta Table Creation
3. ACID Transactions
4. Time Travel
5. Schema Enforcement
6. Schema Evolution
7. Delta Lake Best Practices

Module 6: Data Ingestion and ETL

1. Batch Data Processing
2. Incremental Data Loads
3. Auto Loader Fundamentals
4. ETL Pipeline Development
5. Data Quality Validation
6. Error Handling
7. Pipeline Monitoring

Module 7: Data Transformation Techniques

1. Data Cleansing
2. Data Enrichment
3. Window Functions
4. Complex Transformations
5. Incremental Processing
6. Performance Considerations
7. Reusable Transformation Patterns

Module 8: Streaming Data Processing

1. Structured Streaming Fundamentals
2. Streaming Sources
3. Streaming Sinks
4. Watermarking
5. Checkpointing
6. Trigger Configuration
7. Streaming Monitoring

Module 9: Data Management and Governance

1. Unity Catalog Fundamentals
2. Catalog Structure
3. Schemas and Tables
4. Data Permissions
5. Governance Best Practices
6. Lineage Overview
7. Security Fundamentals

Module 10: Workflow Orchestration

1. Databricks Jobs
2. Task Dependencies
3. Scheduling Workflows
4. Notifications and Alerts
5. Monitoring Executions
6. Troubleshooting Jobs
7. Operational Best Practices

Module 11: Performance Optimization

1. Partitioning Strategies
2. File Management
3. Caching Techniques
4. Query Optimization
5. Delta Optimization Commands
6. Performance Monitoring
7. Cost Optimization

Module 12: Certification Exam Preparation

1. Certification Exam Overview
2. Exam Domains Review
3. Sample Questions Analysis
4. Scenario-Based Exercises
5. Practice Tests
6. Exam Strategies
7. Final Review Session

Laboratórios Práticos

Lab 1: Creating and Managing Databricks Clusters

• Cluster Deployment
• Runtime Configuration
• Resource Management

Lab 2: Building Data Pipelines

• Data Ingestion
• Data Transformation
• Data Validation

Lab 3: Delta Lake Implementation

• Delta Table Creation
• Time Travel Operations
• Schema Evolution

Lab 4: Streaming Data Pipeline

• Structured Streaming Setup
• Checkpoint Configuration
• Stream Monitoring

Lab 5: Workflow Automation

• Job Creation
• Scheduling
• Dependency Management

Lab 6: Performance Tuning

• Query Optimization
• Data Layout Optimization
• Cluster Performance Analysis

Lab 7: End-to-End Data Engineering Project

• Ingest Raw Data
• Build Bronze Layer
• Build Silver Layer
• Build Gold Layer
• Create Automated Workflows
• Implement Data Governance
• Monitor and Optimize Performance

Lab 8: Certification Mock Exam

• Simulado Completo
• Correção Comentada
• Revisão dos Tópicos Críticos
• Estratégias para Aprovação no Exame Oficial Databricks Certified Data Engineer Associate.