Big Data Analysis with Scala and Spark
date_range Starts on March 13, 2017
event_note End date April 10, 2017
list 4 sequences
assignment Level : Introductive
label Computer science
chat_bubble_outline Language : English
card_giftcard 0 point
- /5
0 reviews

Key informations

credit_card Free access

About the content

Manipulating big data distributed over a cluster using functional concepts is rampant in industry, and is arguably one of the first widespread industrial uses of functional ideas. This is evidenced by the popularity of MapReduce and Hadoop, and most recently Apache Spark, a fast, in-memory distributed collections framework written in Scala. In this course, we'll see how the data parallel paradigm can be extended to the distributed case, using Spark throughout. We'll cover Spark's programming model in detail, being careful to understand how and when it differs from familiar programming models, like shared-memory parallel collections or sequential Scala collections. Through hands-on examples in Spark and Scala, we'll learn when important issues related to distribution like latency and network communication should be considered and how they can be addressed effectively for improved performance. Learning Outcomes. By the end of this course you will be able to: - read data from persistent storage and load it into Apache Spark, - manipulate data with Spark and Scala, - express algorithms for data analysis in a functional style, - recognize how to avoid shuffles and recomputation in Spark, Recommended background: You should have at least one year programming experience. Proficiency with Java or C# is ideal, but experience with other languages such as C/C++, Python, Javascript or Ruby is also sufficient. You should have some familiarity using the command line. This course is intended to be taken after Parallel Programming:

more_horiz See plus
more_horiz See less


  • Week 1 - Getting Started + Spark Basics
    Get up and running with Scala on your computer. Complete an example assignment to familiarize yourself with our unique way of submitting assignments. In this week, we'll bridge the gap between data parallelism in the shared memory scenario (learned in the Para...
  • Week 2 - Reduction Operations & Distributed Key-Value Pairs
    This week, we'll look at a special kind of RDD called pair RDDs. With this specialized kind of RDD in hand, we'll cover essential operations on large data sets, such as reductions and joins.
  • Week 3 - Partitioning and Shuffling
    This week we'll look at some of the performance implications of using operations like joins. Is it possible to get the same result without having to pay for the overhead of moving data over the network? We'll answer this question by delving into how we can par...
  • Week 4 - Structured data: SQL, Dataframes, and Datasets
    With our newfound understanding of the cost of data movement in a Spark job, and some experience optimizing jobs for data locality last week, this week we'll focus on how we can more easily achieve similar optimizations. Can structured data help us? We'll look...


  • Dr. Heather Miller, Research Scientist

Content designer

The École polytechnique fédérale de Lausanne (EPFL, English: Swiss Federal Institute of Technology in Lausanne) is a research university in Lausanne, Switzerland, that specialises in physical sciences and engineering.

One of the two Swiss Federal Institutes of Technology, the school was founded by the Swiss Federal Government with the stated mission to:

Educate engineers and scientists to the highest international standing
Be a national center of excellence in science and technology
Provide a hub for interaction between the scientific community and the industry
EPFL is considered one of the most prestigious universities in the world for engineering and sciences, ranking 17th overall and 10th in engineering in the 2015 QS World University Rankings; 34th overall and 12th in engineering in the 2015 Times Higher Education World University Rankings.



Coursera is a digital company offering massive open online course founded by computer teachers Andrew Ng and Daphne Koller Stanford University, located in Mountain View, California. 

Coursera works with top universities and organizations to make some of their courses available online, and offers courses in many subjects, including: physics, engineering, humanities, medicine, biology, social sciences, mathematics, business, computer science, digital marketing, data science, and other subjects.

What is your opinion on this resource ?

You may be interested by...