Conclusion

Google Cloud Dataproc is a managed service for running Apache Hadoop and Spark jobs. It can be used for big data processing and machine learning.

But you could run these data processing frameworks on Compute Engine instances, so what does Dataproc do for you? Dataproc actually uses Compute Engine instances under the hood, but it takes care of the management details for you. It’s a layer on top that makes it easy to spin up and down clusters as you need them.

Learning Objectives

• Explain the relationship between Dataproc, key components of the Hadoop ecosystem, and related GCP services
• Create, customize, monitor, and scale Dataproc clusters
• Run data processing jobs on Dataproc
• Apply access control to Dataproc

Intended Audience

• Data professionals
• People studying for the Google Professional Data Engineer exam

Prerequisites

• Hadoop or Spark experience (recommended)
• Google Cloud Platform account (sign up for free trial at https://cloud.google.com/free if you don’t have an account)

This Lesson Includes

• 49 minutes of high-definition video
• Many hands-on demos
   

The github repository is at https://github.com/cloudacademy/dataproc-intro.