Hello, and welcome to data engineering on Microsoft Azure. The focus of this course is to prepare you for Microsoft's DP-203 exam. My name's Guy Hummel and I'm a Microsoft certified Azure solutions architect and data engineer. The DP-203 exam tests your knowledge in three subject areas. Designing and implementing data storage, developing data processing and securing, monitoring and optimizing data storage and data processing. I'm not going to talk about every item in the exam guide, but I'll go over some of the highlights of what you'll need to know.

By far the most important service you need to know for the exam is Azure Synapse Analytics. This service allows you to store and analyze huge amounts of data. Microsoft has made it the central service in its data analytics product line. An alternative analytics service is Azure Databricks, which is a collaboration between Microsoft and Databricks, the company founded by the people who created Apache Spark. Naturally, Azure Databricks uses Spark, but you can also use Spark on Azure Synapse Analytics, so the two services are effectively competitors. Although both of these services have their own storage capabilities, it's common to use another service called Azure Data Lake Storage Gen2 to store their data. It's essentially a layer on top of Azure Blob storage. This layer adds features that make it especially useful for big data processing.

The exam tests your knowledge of these three services in every section. The first section is about designing and implementing data storage. First, to make queries as fast and efficient as possible, you need to know how to partition a data store into multiple shards using any of the three services I mentioned earlier. Second, you need to know how to design and implement the data exploration layer, primarily using Synapse Analytics and Microsoft Purview, which is a service for unifying and managing pockets of data.

The next section of the exam guide is about developing data processing solutions. This includes ingesting and transforming data, batch processing, stream processing and managing batches and pipelines. You need to know how to transform data using both Spark and SQL and you need to be able to use the transformation features of Data Factory, Azure Synapse Pipelines, and Stream Analytics. Data Factory is a data integration service. It lets you build pipelines that can take data from one service, such as Azure Storage, transform it, and store it in another service, such as Azure Databricks. Azure Synapse pipelines is essentially a subset of Data Factory that has been bolted onto Azure Synapse Analytics. So, if you know how to use Data Factory, you'll know how to use Synapse Pipelines too. They both support Spark and SQL based transformations. Azure Stream Analytics is an older service that uses a SQL like query language.

For batch processing, you need to know to do it using four different services, Data Factory, Azure Synapse Pipelines, Azure Data Bricks and Data Lake Storage Gen2. For stream processing, you need to be able to use Stream Analytics, a service that's focussed entirely on stream processing, Azure Databricks and Azure Event Hubs, which is a service that can ingest massive amounts of steaming data and pass it onto either Stream Analytics or Azure Databricks. Finally, you need to be able to manage batches and pipelines using Data Factory or Synapse Pipelines, which again, are very similar to each other.

The final section of the exam guide is about how to secure, monitor and optimize data storage and data processing. The security topic is not just about restricting access to data, which is very important, of course, but also protecting sensitive information using techniques such as applying data masking to credit card numbers or encrypting an entire database. The most important service for the monitoring part of this section is, not surprisingly, Azure Monitor, which you can use to monitor and configure alerts for almost every other Azure service. The optimization subsection doesn't include new services. Instead, you need to know how to optimize the performance of services like Synapse Analytics, Data Factory and Azure Databricks.

Please note that this course doesn't cover the topics in the exact same order as the exam guide because there's so much overlap between the different sections. For example, Azure Databricks is in every section of the exam guide, but we don't have separate Azure Databricks lessons for every section. Now are you ready to learn about Azure Data Engineering? Then let's get started, but please remember to rate this introduction before you go on to the next lesson, thanks.

DP-203 Exam Prep - Introduction