Spark入门教程收集

最新推荐文章于 2025-11-24 22:12:38 发布

原创最新推荐文章于 2025-11-24 22:12:38 发布 · 59 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #ajax #大数据 #分布式 #前端

本文介绍了Apache Spark的快速入门指南，包括其作为开源集群计算框架的优势，如处理实时生成数据的能力，以及与Hadoop MapReduce相比在内存中运行的优化特性。详细探讨了Spark的核心抽象——弹性分布式数据集(RDD)和有向无环图(DAG)，并通过示例展示了如何使用Spark进行数据过滤操作。此外，还提供了丰富的资源链接，涵盖Spark SQL、流处理、机器学习等方面，适合初学者快速掌握Spark的基本使用。

https://spark.apache.org/ 官网

Spark教程

https://www.yiibai.com/spark/apache-spark-filter-function.html

Apache Spark是一个开源集群计算框架。其主要目的是处理实时生成的数据。

Spark建立在Hadoop MapReduce的顶部。它被优化为在内存中运行，而Hadoop的MapReduce等替代方法将数据写入计算机硬盘驱动器或从计算机硬盘驱动器写入数据。因此，Spark比其他替代方案更快地处理数据。

Spark架构依赖于两个抽象：

弹性分布式数据集(RDD)
有向无环图(DAG)

scala> val data = sc.parallelize(List(10,20,30))
data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[2] at parallelize at <console>:24

scala> data.collect
res3: Array[Int] = Array(10, 20, 30)

scala> val abc = data.filter(x => x!=30)

Spark 编程指南

https://www.w3cschool.cn/spark/

Spark SQL教程

https://www.w3cschool.cn/spark_sql/

Spark Overview

https://spark.apache.org/docs/3.0.0/index.html

Programming Guides:

Quick Start: a quick introduction to the Spark API; start here!
RDD Programming Guide: overview of Spark basics - RDDs (core but old API), accumulators, and broadcast variables
Spark SQL, Datasets, and DataFrames: processing structured data with relational queries (newer API than RDDs)
Structured Streaming: processing structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams)
Spark Streaming: processing data streams using DStreams (old API)
MLlib: applying machine learning algorithms
GraphX: processing graphs

API Docs:

Quick Start

http://spark.apache.org/docs/latest/quick-start.html

RDD Programming Guide

http://spark.apache.org/docs/latest/rdd-programming-guide.html

Spark 3.0.1 ScalaDoc

https://spark.apache.org/docs/3.0.0/api/scala/org/apache/spark/index.html

Apache Spark Examples

https://spark.apache.org/examples.html

Additional Examples

Many additional examples are distributed with Spark: