Conquer Big Data through Spark

Course Background:

Apache Spark™ is a fast and general engine for large-scaledata processing. Spark has an advanced DAGexecution engine that supports cyclic data flow and in-memory computing. You can run programs up to 100x faster than HadoopMapReduce in memory, or 10x faster on disk.:

Sparkpowers a stack of high-level tools including SparkSQL,MLlib for machine learning, GraphX, and Spark Streaming. You can combine these libraries seamlessly in the sameapplication:

You canrun Spark readily using its standalone cluster mode, on EC2, or run it on Hadoop YARN or Apache Mesos. It can readfrom HDFS, HBase, Cassandra, and any Hadoop data source:

Write applications quickly in Java, Scala orPython.Spark offers over 80 high-level operators thatmake it easy to build parallel apps. And you can use it interactively from the Scala and Python shells.

ApacheSpark has seen phenomenal adoption, being widely slated as the successor toHadoop MapReduce, and being deployed in clusters from a handful to thousands ofnodes.

In the past few years ,Databricks, with the help of the Spark community, hascontributed many improvements to Apache Spark to improve its performance,stability, and scalability. This enabled Databricks to use Apache Spark to sort100 TB of data on 206 machines in 23 minutes, which is 3X faster than theprevious Hadoop 100TB result on 2100 machines. Similarly, Databricks sorted 1PB of data on 190 machines in less than 4 hours, which is over 4X faster thanthe previous Hadoop 1PB result on 3800 machines.

 

 Spark is fulfilling its promise to serve as a faster and morescalable engine for data processing of all sizes. Spark enables equallydramatic improvements in time and cost for all Big Data users.

 

Course Introduction:

This course almost covers everything for ApplicationDeveloper to build diverse Spark applications to fulfill all kinds of businessrequirements: Architecture of Spark、the programming model in Spark、internalsof Spark、Spark SQL、MLlib、GraphX、Spark Streaming、Testing、Tuning、Spark onYarn、JobServer and SparkR.

Additional，this course also covers the very necessaryskills you need to write Scala code in Spark, to help whom is not familiar withScala.

 

Who Needsto Attend

Anyone whois interested in Big Data Development;

HadoopDeveloper；

Other BigData Developer;

 

Prerequisites

   Be familiar with the basics of object-oriented programming;

CourseOutline

 

Day 1	Class 1：The architecture of Spark 1 Ecosystem of Spark 2 Design of Spark 3 RDD 4  Fault-tolerance in Spark   Class 2Programming with Scala 1 Classes and Objects in Scala 2 Funtional Object 3 Traits 4 Case class and Pattern Matching 5 Collections 6 Implicit Conversions and Parameters 7 Actors and Concurrency   Class 3：Spark Programming Model 1 RDD 2 transformation 3 action 4 lineage 5 Dependency   Class 4：Spark Internals 1 Spark Cluster 2 Job Scheduling 3 DAGScheduler 4 TaskScheduler 5 Task Internal
TIME	CONTENT	Note
Day 2	Class 5：Broadcasts and Accumulators 1  Broadcast Internal 2  Best practice in Broadcast 3  Accumulators Internal 4  Best practice in Accumulators   Class 6：Action in programming Spark 1 Data Source：File、HDFS、HBase、S3; 2 IDEA 3 Maven 4 sbt. 5 Code 6 Deployment   Class 7：Deep in Spark Driver 1 The Secret of SparkContext 2 The Secret of  SparkConf 4 The Secret of  SparkEnv   Class 8：Deep in RDD 1 DAG 2 Scala RDD Function 3 Spark Java RDD Function 4 RDD Tuning

 

TIME	CONTENT	NOTE
Day 3	Class 9：Machine Learning on Spark 1 LinearRegression 2 K-Means 3 Collaborative Filtering   Class 10: Graph Computation on Spark 1 Table Operators 2 Graph Operators 3 GraphX Algorithms   Class 11: Spark SQL 1 Parquet、JSON、JDBC 2 DSL 3 SQL on RDD   Class 12：Spark Streaming 1 DStream 2 transformation 3 checkpoint 4 Tuning
TIME	CONTENT	NOTE
Day 4	Class 13：Spark on Yarn 1 Internals of Spark on Yarn 2 Best practice of Spark on Yarn   Class 14：JobServer 1 Restful Architecture of JobServer 2 JobServer APIs 3 Best Practice of JobServer   Class 15：SparkR 1 Programming in R 2 R on Spark 3 Internals of SparkR 4 SparkR API   Class 16：Spark Tuing 1 Logs 2 Concurency 3 Memory 4 GC 5 Serializers 6 Safety 7 14s cases of Tuning