What you'll learn

Process data as it arrives
Handle data of any scale in real time
Use Structured Streaming and SQL on streaming data
Use low-level, high-control APIs with DStreams
Integrate Spark with Kafka, JDBC, Cassandra
Master advanced concepts including window functions and watermarking

Requirements

Scala and functional programming essentials, at the level of the beginners course (implicits are a plus)
Spark fundamentals, including Spark SQL, DataFrames and RDDs
Good general knowledge of parallel/distributed systems (nodes, threads, etc)

Description

Updated for Spark 3.0

In this course, we will learn how to stream big data with Apache Spark 3. You'll write 1500+ lines of Spark code yourself, with guidance, and you will become a rockstar.

This course is for Spark & Scala programmers who now need to work with streaming data, or who need to process data in real time.

Why Spark in Scala:

it's blazing fast for big data

its demand has exploded

it's a highly marketable skill

it's well maintained, with dozens of high-quality extensions

it's a foundation for a data scientist

I like to get to the point and get things done. This course

deconstructs all concepts into the critical pieces you need

selects the most important ideas and separates them into what's simple but critical and what's powerful

sequences ideas in a way that "clicks" and makes sense throughout the process of learning

applies everything in live code

The end benefits are still much greater:

a completely new mental model around data streaming

significantly more marketable resume

more enjoyable work - Spark is fun!

This course is for established programmers with experience with both Scala and Spark at least at the level of the Rock the JVM essential courses for Scala and Spark. I already assume a solid understanding of general programming fundamentals.

This course is NOT for you if

you've never written Scala or Spark code before

you don't have some essential parallel programming background (e.g. what's a process/a thread)

The course is comprehensive, but you'll always see me get straight to the point. So make sure you have a good level of focus and commitment to become a badass programmer.

I believe both theory and practice are important. That's why you'll get lectures with code examples, real life code demos and assignments, plus additional resources, instructions, exercises and solutions. At the end of the course, you'll have written thousands of lines of Spark.

I've seen that my students are most successful - and my best students work at Google-class companies - when they're guided, but not being told what to do. I have exercises waiting for you, where I offer my (opinionated) guidance but otherwise freedom to experiment and improve upon your code.

Definitely not least, my students are most successful when they have fun along the way!

So join me in this course and let's rock the JVM!
Who this course is for:

Scala & Spark programmers who need to set up data pipelines in real time
Data scientists who need to work with streaming data

Screenshots