Spark学习之 Spark SQL( 一 )

最新推荐文章于 2024-08-09 12:09:29 发布

Crzayliyang-架构Young

最新推荐文章于 2024-08-09 12:09:29 发布

阅读量279

点赞数

CC 4.0 BY-SA版权

文章标签：大数据SparkSQL Spark 大数据

本文链接：https://blog.youkuaiyun.com/Crazy_liyang/article/details/82149830

本文是作者学习记录，先创建people.json文件，上传到HDFS文件系统指定目录，查看文件完整性，接着启动Spark Shell并执行代码，最后进行测试。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

感谢大牛的系列文章, 本文只是本人学习过程的记录, 首先向大神致敬!

https://blog.youkuaiyun.com/lovehuangjiaju/article/details/48661847

1.创建文件people.json

{"name":"Michael", "age":27}
{"name":"Andy",    "age":30}
{"name":"Justin",  "age":19}

2.上传到HDFS文件系统目录位置: /data/people.json

hdfs dfs -put ./people.json /data

3.在HDFS中查看文件是否完整,如下

[root@hd-02 ~]# hdfs dfs -cat /data/people.json
{"name":"Michael"}
{"name":"Andy", "age":30}
{"name":"Justin", "age":19}
[root@hd-02 ~]#

4.启动Spark Shell , 并执行如下代码:

bin/spark-shell
scala> val sqlContext = new org.apache.spark.sql.SQLContext(sc)
scala> val df = sqlContext.read.json("hdfs://hd-01:9000/data/people.json")

5.测试

scala> df.show
+----+-------+
| age|   name|
+----+-------+
|null|Michael|
|  30|   Andy|
|  19| Justin|
+----+-------+

scala> df.printSchema()
root
 |-- age: long (nullable = true)
 |-- name: string (nullable = true)

scala> df.select("name").show
+-------+
|   name|
+-------+
|Michael|
|   Andy|
| Justin|
+-------+

scala> df.filter( df("age") > 21  ).show
+---+----+
|age|name|
+---+----+
| 30|Andy|
+---+----+
scala> df.registerTempTable("people")

scala> val teenagers = sqlContext.sql("SELECT name, age FROM people WHERE age >= 13 AND age <= 19")
teenagers: org.apache.spark.sql.DataFrame = [name: string, age: bigint]

scala> teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
Name: Justin