Course Description

Course Overview

The Data Engineering on Google Cloud Platform (DEGCP) course is designed to provide individuals with the knowledge and skills necessary to design and build scalable data processing systems on the Google Cloud Platform (GCP). This course focuses on the key data engineering concepts, tools, and best practices for ingesting, processing, and analyzing data on GCP.

Prerequisites

To enroll in the DEGCP course, participants should have a strong understanding of data engineering principles and experience with at least one programming language. Familiarity with cloud computing concepts and GCP fundamentals will be beneficial. Participants should also have access to a GCP project or demo environment to practice the concepts covered in the course.

Methodology

The DEGCP course follows a blended learning approach, combining theoretical instruction, demonstrations, discussions, and hands-on labs. Participants will engage in instructor-led sessions where data engineering concepts, best practices, and GCP services are explained. They will also have access to GCP resources and tools to gain practical experience in building data processing systems. The course encourages active participation, discussions, and collaborative problem-solving to reinforce learning.

Course Outline

Introduction to Data Engineering on GCP

Overview of data engineering concepts and challenges

Understanding the benefits of data engineering on GCP

Exploring GCP data engineering services and tools

Ingesting and Transforming Data

Implementing data ingestion pipelines using Cloud Storage and Cloud Pub/Sub

Transforming data with Cloud Dataflow and Cloud Dataprep

Applying data quality and cleaning techniques

Storing and Processing Data

Utilizing BigQuery for storing and querying large datasets

Building batch and stream processing pipelines with Cloud Dataflow

Implementing data partitioning and clustering strategies

Data Warehousing and Analytics

Designing and building data warehousing solutions with BigQuery

Implementing data modeling and optimization techniques

Utilizing BI tools and visualizations for data analysis

Real-Time Data Processing

Implementing real-time data processing pipelines with Cloud Dataflow and Cloud Pub/Sub

Performing windowing and aggregations in real-time scenarios

Handling late data and out-of-order events in real-time processing

Data Orchestration and Workflow Management

Managing data pipelines and workflows with Cloud Composer

Utilizing Apache Airflow for workflow scheduling and dependency management

Implementing error handling and retries in data pipelines

Outcome

By the end of the DEGCP course, participants will have:

• Developed a comprehensive understanding of data engineering concepts and best practices on GCP
• Acquired practical knowledge in ingesting, transforming, and processing data on GCP
• Gained expertise in utilizing GCP data engineering services such as BigQuery, Cloud Dataflow, and Cloud Pub/Sub
• Learned techniques for designing scalable and efficient data processing pipelines
• Gained hands-on experience through practical labs and exercises
• Prepared to design and build scalable data processing systems on GCP as a Data Engineer

Labs

The DEGCP course includes hands-on labs that provide participants with practical experience in building data processing systems on GCP. Some examples of lab exercises include:

• Ingesting data from various sources into GCP using Cloud Storage and Cloud Pub/Sub
• Transforming and cleaning data using Cloud Dataflow and Cloud Dataprep
• Building batch processing pipelines with BigQuery and Cloud Dataflow
• Implementing real-time data processing pipelines using Cloud Dataflow and Cloud Pub/Sub
• Designing and building data warehousing solutions with BigQuery
• Managing data pipelines and workflows with Cloud Composer

These labs enable participants to apply the concepts learned in the course and gain hands-on experience in building scalable data processing systems on GCP, allowing them to develop practical skills as a Data Engineer on Google Cloud Platform.