PySpark
Master Spark Structured Streaming using Python (PySpark) on Azure Databricks Cloud with a end-to-end Project
Lo que serás capaz de hacer
About the Course I am creating Apache Spark and Databricks - Stream Processing in Lakehouse using the Python Language and PySpark API. This course will help you understand Real-time Stream processing using Apache Spark and Databricks Cloud and apply that knowledge to build real-time stream processing solutions . This course is example-driven and follows a working session-like approach. We will take a live coding approach and explain all the needed concepts. Capstone Project This course also includes an End-To-End Capstone project. The project will help you understand the real-life project design, coding, implementation, testing, and CI/CD approach. Who should take this Course? I designed this course for software engineers willing to develop a Real-time Stream Processing Pipeline and application using Apache Spark. I am also creating this course for data architects and data engineers who are responsible for designing and building the organization’s data-centric infrastructure. Another group of people is the managers and architects who do not directly work with Spark implementation. Still, they work with those implementing Apache Spark at the ground level. Spark Version used in the Course. This Course is using the Apache Spark 3.5 . I have tested all the source code and examples used in this Course on Azure Databricks Cloud using Databricks Runtime 14.1 .
- Real-time Stream Processing Concepts
- Spark Structured Streaming APIs and Architecture
- Working with Streaming Sources and Sinks
- Kafka for Data Engineers
- Working With Kafka Source and Integrating Spark with Kafka
- State-less and State-full Streaming Transformations
Temario
UNIDAD 1 - Before you start · 4 leccionesUNIDAD 1 - Before you start
UNIDAD 1 - Before you start
- About the Course
- Course Prerequisite
- Source Code and Other Resources
- Note for Students - Before Start
UNIDAD 2 - Setup your environment · 3 leccionesUNIDAD 2 - Setup your environment
UNIDAD 2 - Setup your environment
- Spark Development Environments
- Setup your Databricks Community Cloud Environment
- Working in Databricks Workspace
UNIDAD 3 - Getting Started with Spark Streaming · 9 leccionesUNIDAD 3 - Getting Started with Spark Streaming
UNIDAD 3 - Getting Started with Spark Streaming
- Batch processing to stream processing
- Your Spark application - Applying Best Practice
- Your first streaming application - Implementing Stream
- Stream Processing Model in Spark
- Create Another Streaming Application
- Stream Triggers
- Incremental Batch Processing
- Streaming Sources and Sinks
- Creating Chain of Streams
UNIDAD 4 - Kafka for Data Engineers · 8 leccionesUNIDAD 4 - Kafka for Data Engineers
UNIDAD 4 - Kafka for Data Engineers
- An Introduction to Kafka
- Creating Kafka Cluster in Cloud
- Kafka Core Concepts
- Producing Data to Kafka Topic
- Consuming Data from Kafka Topic
- Working with Kafka Topic Data
- How to Implement Idempotence
- Working with Kafka Sink
UNIDAD 5 - Streaming Aggregates and State Management · 8 leccionesUNIDAD 5 - Streaming Aggregates and State Management
UNIDAD 5 - Streaming Aggregates and State Management
- Streaming Aggregates and State Store
- Incremental Aggregates and Update Mode
- Spark Streaming Output Modes
- Statefull Vs Stateless Aggregation
- Implementing Stateless Streaming Aggregation
- Timebound Stateful Tumbling Window Aggregation
- Watermarking and State Store Cleanup
- Sliding Window Aggregates
UNIDAD 6 - Working with Databricks Platform · 14 leccionesUNIDAD 6 - Working with Databricks Platform
UNIDAD 6 - Working with Databricks Platform
- Introduction to Databricks
- Creating Azure Free Account
- Azure Portal Overview
- Creating Azure Databricks Service
- Introduction to Azure Databricks Workspace
- Azure Databricks Architecture
- Creating Azure Databricks Cluster
- Introduction to Databricks Notebooks
- Notebooks Magic Commands and Utilities
- Databricks Notebooks Utilities
- Introduction to Databricks Unity Catalog
- Introduction to Databricks Workflow Jobs
- Introduction to Databricks Rest API
- Introduction to Databricks CLI
UNIDAD 7 - Capstone Project - Implementing Real-time Project in Lakehouse · 30 leccionesUNIDAD 7 - Capstone Project - Implementing Real-time Project in Lakehouse
UNIDAD 7 - Capstone Project - Implementing Real-time Project in Lakehouse
- Project Scope and Background
- Taking out the operational requirement
- Storage Design
- Implement Data Security
- Implement Resource Policies
- Decouple Data Ingestion
- Design Bronze Layer
- Design Silver and Gold Layer
- Setup your source control
- Setup your environment
- Create a development workspace
- Create and Configure Storage Layer
- Create Unity Catalog Metastore
- Create Catalog and External Locations
- Start Coding
- Test your code
- Load historical data
- Ingest into bronze layer
- Process the silver layer
- Handling multiple updates
- Implementing Gold Layer
- Creating a run script
- Preparing for Integration testing
- Creating Test Data Producer
- Creating Integration Test for Batch mode
- Creating Integration Test for Stream mode
- Implementing CI CD Pipeline
- Develop Build Pipeline
- Develop Release Pipeline
- Creating Databricks CLI Script
UNIDAD 8 - Final Word · 1 leccionesUNIDAD 8 - Final Word
UNIDAD 8 - Final Word
- An End is a new Beginning
UNIDAD 9 - Archived - Old Course Content · 31 leccionesUNIDAD 9 - Archived - Old Course Content
UNIDAD 9 - Archived - Old Course Content
- Spark Development Environment
- Windows User - Spark Installation Prerequisites
- Windows User - Installing Apache Spark
- Windows User - Setup and test your IDE
- Mac User - Installing Apache Spark
- Mac User - Setup and test your IDE
- Install and run Apache Kafka
- Stream processing model in Spark
- Working with Files and Directories
- Streaming Sources, Sinks and Output Mode
- Fault Tolerance and Restarts
- Introduction to Stream Processing
- Spark Streaming APIs - DStream Vs Structured Streaming
- Creating your first stream processing application
- Streaming from Kafka as a Source
- Working with Kafka Sinks
- Multi-query Streams Application
- Kafka Serialization and Deserialization for Spark
- Creating Kafka AVRO Sinks
- Working with Kafka AVRO Source
- Stateless Vs Statefull transformations
- Event time and Windowing
- Tumbling Window aggregate
- Watermarking your windows
- Watermark and output modes
- Sliding Window
- Joining Stream to static source
- Joining Stream to another Stream
- Streaming Watermark
- Streaming Outer Joins
- Final Word
Tu certificado lo emite la SEP, no solo una plataforma
Al demostrar la competencia recibes un certificado emitido por la Secretaría de Educación Pública a través del Gobierno del Estado de México: un documento oficial que puedes presentar con confianza ante cualquier empleador o institución.
Tu certificado con INEC
- Lo emite la Secretaría de Educación Pública (SEP), a través del Gobierno del Estado de México
- Documento oficial con folio verificable en línea
- Lo otorga la Universidad Tecnológica Fidel Velázquez
Otras plataformas en línea
- Constancia privada emitida por la misma plataforma
- Solo acredita horas cursadas o asistencia
- Sin respaldo de una autoridad educativa
Tu itinerario, de la inscripción al certificado
- 1
Inscripción con pago único
Pagas $2,800 MXN una sola vez y entras de inmediato al curso.
- 2
Estudias a tu ritmo
9 unidades con videos, lecturas y actividades, disponibles 24/7 durante 12 meses.
- 3
Cuestionarios por unidad
Cuestionarios para comprobar lo aprendido.
- 4
Certificado
Emitido por la SEP a través del Gobierno del Estado de México, con folio verificable.
Un certificado oficial de la SEP
Muestra · verificable por folio y QR
- Emitido por la Secretaría de Educación Pública (SEP), a través del Gobierno del Estado de México. No es una constancia privada: es un documento oficial, y ese respaldo marca la diferencia frente a otras plataformas en línea.
- Contenido impartido por Prashant Kumar Pandey.
- Folio y QR verificables para compartirlo con quien te contrate.
Empieza hoy, a tu ritmo
Un solo pago y acceso inmediato al curso completo durante 12 meses.
- 9 unidades · 108 lecciones · 45 h
- 12 meses de acceso
- Certificado emitido por la SEP (Gob. del Estado de México) con folio verificable
Sobre tu inscripción
¿Cómo es el pago?
Un solo pago de $2,800 MXN. No hay mensualidades ni cargos recurrentes.
¿Cuánto tiempo tengo acceso?
12 meses desde tu inscripción, 24/7, para avanzar a tu ritmo.
¿Quién imparte el contenido?
El curso es impartido por Prashant Kumar Pandey; INEC gestiona tu inscripción, evaluación y certificado.
¿Qué pasa cuando vence mi acceso?
Puedes renovar tu cuenta tú mismo desde la plataforma, sin trámites ni esperas.
