Confluent Kafka 101 Learning Reference

This repository is a public learning reference for the Confluent Developer Apache Kafka 101 course.

The aim is not to copy the course transcript. The aim is to document my own understanding of Apache Kafka, Confluent Cloud, Schema Registry, Kafka Connect, Flink SQL, and the practical CLI issues I encountered while following the course.

Course reference: https://developer.confluent.io/courses/apache-kafka/events/

Repository Contents

File	Purpose
docs/confluent_kafka_101_notes.md	Main course notes, structured by module
docs/confluent_kafka_101_cli_commands.md	CLI-only command reference used during the exercises
docs/confluent_kafka_101_troubleshooting.md	Consolidated issue list and detailed fixes
docs/confluent_kafka_101_setup_windows.md	Windows setup notes for the course exercises
docs/confluent_kafka_101_setup_macos.md	macOS setup notes for the course exercises
docs/event_notes_data_streaming_meetup_2026_06_11.md	Notes from the streaming meetup attended during the learning journey
docs/real_time_streaming_architecture.md	Architecture notes on Kafka, Flink, Snowflake, dbt, and Iceberg
diagrams/confluent_kafka_101_complete_platform.mmd	One high-level diagram showing producers, consumers, brokers, partitions, Schema Registry, Kafka Connect, and Flink
diagrams/kafka_hot_iceberg_cold_retention.mmd	Hot and cold retention pattern diagram
diagrams/kafka_realtime_and_warehouse_lanes.mmd	Real-time and warehouse delivery lane diagram
config/config.example.ini	Safe example config file with placeholders only
examples/consumer.py	Course-style Python consumer example with no secrets

Security Note

This repository must not contain real API keys, API secrets, OAuth callback URLs, Confluent local credential files, unredacted screenshots, or real config.ini files.

Only placeholder values such as <API_KEY>, <API_SECRET>, <CLUSTER_ID>, and <SCHEMA_REGISTRY_ENDPOINT> should be used.

What Code Is Actually Needed

Only a small amount of code was needed for the course exercises.

• examples/consumer.py is the only Python script worth keeping.
• config/config.example.ini is worth keeping as a safe template.
• Most of the hands-on work was done with the Confluent CLI and the Confluent Cloud UI.
• Flink SQL statements are documented in the main notes rather than in the CLI-only command file.

Learning Scope

The notes cover:

• Kafka topics, partitions, brokers, replication, producers, and consumers
• Offsets, consumer groups, replay, and retention
• Confluent Cloud CLI setup and usage
• Python consumer setup
• Schema Registry and Avro deserialization
• Kafka Connect and source connectors
• Flink SQL for stream processing
• Real-time streaming architecture patterns
• Kafka hot storage and Iceberg cold storage patterns
• Practical troubleshooting from hands-on exercises

High-Level Mental Model

Kafka is a durable event log. Producers write events into topics. Consumers read those events independently. Reading an event does not remove it from Kafka.

Kafka is useful when data needs to move continuously, multiple systems need the same events, replay matters, or event-driven applications are needed.

Kafka is not automatically the right answer for every data pipeline. If the only requirement is a simple daily extract into a warehouse, a simpler batch pipeline may be more appropriate.