Visão Geral

Este Curso Introduction to Kafka for C# Developers, ensina os participantes como usar a plataforma de streaming de eventos Apache Kafka para pipelines de dados de alto desempenho, análise de streaming, integração de dados e aplicativos de missão crítica. O .NET Core é usado como estrutura subjacente.

Objetivo

Após realizar este Curso Introduction to Kafka for C# Developers, você será capaz de:

• Entenda o uso do Kafka para mensagens de alto desempenho
• Identifique os usos do Kafka em microsserviços
• Explique os benefícios dos padrões Kafka
• Diferencie entre mensagens e corretores de mensagens
• Descrever os ambientes de mensagens Kafka
• Desenvolver produtores e consumidores para Kafka
• Reconheça como o Kafka permite aplicativos nativos da nuvem
• Resuma as características e arquitetura de Kafka
• Demonstrar como processar e consumir mensagens do Kafka usando .NET Core Web API, MVC e Worker (BackgroundService)
• Demonstrar o papel de Kafka no projeto ponta a ponta envolvendo frontend .NET MVC, backend .NET Web API, .NET Worker, banco de dados SQL do Azure e cache Redis
• Projete sistemas distribuídos de alto rendimento baseados em Kafka
• Descreva o particionamento, replicação e tolerância a falhas integrados do Kafka

Pre-Requisitos

• Ter um conhecimento básico de mensagens, nuvem, desenvolvimento, arquitetura e virtualização é benéfico.
• É necessária experiência no desenvolvimento de aplicativos .NET com C#. Recomenda-se experiência anterior com .NET Core.

Materiais

Conteúdo Programatico

Introduction to Kafka

1. Messaging Architectures
2. What is Kafka?
3. When to Use Kafka?
4. Kafka Architecture
5. Core concepts in Kafka
6. Why Kafka Cluster?
7. Sample Multi-Broker Cluster
8. Overview of ZooKeeper
9. Kafka Cluster & ZooKeeper
10. Schema Registry
11. Who Uses Kafka?

The Inner Workings of Apache Kafka

1. A Kafka Cluster High-Level Interaction Diagram
2. Topics & Partitions
3. The Terms Event/Message/Record
4. Message Offset
5. Message Retention Settings
6. Deleting Messages
7. The Flush Policies
8. Writing to Partitions
9. Batches
10. Batch Compression
11. Partitions as a Unit of Parallelism
12. Message Ordering
13. Kafka Default Partitioner
14. The Load Balancing Aspect
15. Kafka Message Production Schematics
16. ZooKeeper
17. Reading from a Topic
18. Consumer Lag
19. Consumer Group
20. Consumer Group Diagram
21. The Broker
22. Broker Hardware Consideration
23. OS and File System
24. The Leader and Followers Pattern
25. Partition Replication Diagram
26. Controlled Shutdown
27. Controlling Message Durability with Minimum In-Sync Replicas
28. Log Compaction
29. Frequent Operational Problems
30. Some Kafka Design FAQs

Using Apache Kafka

1. What is Confluent?
2. Confluent Cloud
3. Confluent Cloud Resource Hierarchy
4. Setting up Confluent Cloud on Azure
5. Setting up Confluent Cloud using Confluent.io
6. Select the Confluent Cloud Cluster Type
7. Choose the Cloud Provider
8. Setting up Confluent Cloud using Azure Marketplace
9. Select Confluent Cloud in Azure Marketplace
10. Purchase Confluent Cloud
11. The Cluster View
12. Exploring the Confluent Cloud Console
13. Topics
14. Topics Advanced Settings
15. Searching for Messages in a Topic
16. The Confluent CLI
17. The confluent CLI Command Examples
18. Kafka Cluster Planning – Producer/Consumer Throughput
19. Managing Topics in Confluent Cloud Console
20. Editing an Existing Topic
21. Delete a Topic
22. Kafka and .NET
23. .NET Kafka Architectures
24. Packages
25. Installing the Packages
26. Navigating .NET Client Documentation
27. Important Classes and Interfaces
28. appsettings.json Kafka Configuration
29. Loading the Configuration from appsettings.json
30. Produce and ProduceAsync Methods
31. Produce vs. ProduceAsync
32. Error Handling
33. Consuming Messages
34. Creating and Deleting Topics
35. Copying Data from Between Environments
36. Mocking Datasets using Datagen Connector
37. Monitoring Confluent Cloud
38. Monitoring Confluent Cloud using cURL
39. Motoring Confluent Cloud using third-party Tools

Building Data Pipelines

1. Building Data Pipelines
2. What to Consider When Building Data Pipelines
3. Timeliness
4. Reliability
5. High and Varying Throughput
6. Evolving Schema
7. Data Formats
8. Protobuf (Protocol Buffers) Overview
9. Avro Overview
10. Avro Schema Example
11. JSON Schema Example
12. Managing Data Evolution Using Schemas
13. Confluent Schema Registry
14. Confluent Schema Registry in a Nutshell
15. Schema Management on Confluent Cloud
16. Create a Schema using Confluent CLI
17. Create a Schema from the Web UI
18. Schema Change and Backward Compatibility
19. Collaborating over Schema Change
20. Handling Unreadable Messages
21. Deleting Data
22. Segregating Public and Private Topics
23. Transformations
24. Security
25. Failure Handling
26. Agility and Coupling
27. Ad-hoc Pipelines
28. Metadata Loss
29. Extreme Processing
30. Kafka Connect vs. Producer and Consumer

Integrating Kafka with Other Systems

1. Introduction to Kafka Integration
2. Kafka Connect
3. Running Kafka Connect Operating Modes
4. Key Configurations for Connect workers:
5. Kafka Connect API
6. Kafka Connect Example – File Source
7. Kafka Connect Example – File Sink

Kafka Security

1. Kafka Security
2. Encryption and Authentication using SSL
3. Configuring Kafka Brokers
4. Authenticating Using SASL
5. Authorization and ACLs
6. Securing a Running Cluster
7. ZooKeeper Authentication

Monitoring Kafka

1. Metrics Basics
2. JVM Monitoring
3. Garbage collection
4. Java OS monitoring
5. OS Monitoring
6. Kafka Broker Metrics
7. Under-Replicated Partitions
8. Active controller count
9. Request handler idle ratio
10. Intelligent Thread Usage
11. All topics bytes in
12. All topics bytes out
13. All topics messages in
14. Partition count
15. Leader count
16. Offline partitions
17. Request metrics
18. Logging
19. Client Monitoring
20. Producer Metrics
21. Overall producer metrics
22. Per-broker and per-topic metrics
23. Consumer Metrics
24. Fetch Manager Metrics
25. Per-broker and per-topic metrics
26. Consumer coordinator metrics
27. Quotas
28. Lag Monitoring
29. End-to-End Monitoring

Apache Kafka Best Practices

1. Partitions
2. Consumers
3. Producers
4. Brokers