1.在pyspark中创建文件流
$ cd /usr/local/spark/mycode
$ mkdir streaming
$ cd streaming
$ mkdir logfile
$ cd logfile
进入pyspark创建文件流。请另外打开一个终端窗口，启动进入pyspark
>>> from pyspark import SparkContext
>>> from pyspark.streaming import
StreamingContext
>>> ssc =
StreamingContext(sc, 10)
>>> lines = ssc. \
... textFileStream('file:///usr/local/spark/mycode/streaming/logfile')
>>> words = lines.flatMap(lambda line: line.split(' '))
>>> wordCounts = words.map(lambda x : (x,1)).reduceByKey(lambda a,b:a+b)
>>> wordCounts.pprint()
>>> ssc.
start()
>>> ssc.awaitTermination()
上面在pyspark中执行的程序，一旦你输入ssc.
start()以后，程序就开始自动进入循环监听状态，屏幕上会显示一堆的信息，如下-------------------------------------------
Time: 2018-12-30 15:35:30
-------------------------------------------
-------------------------------------------
Time: 2018-12-30 15:35:40
-------------------------------------------
-------------------------------------------
Time: 2018-12-30 15:35:50
-------------------------------------------
在“/usr/local/spark/mycode/streaming/logfile”目录下新建一个log.txt文件，就可以在监听窗口中显示词频统计结果2. 采用独立应用程序方式创建文件流$ cd /usr/local/spark/mycode
$ cd streaming
$ cd logfile
$ vim FileStreaming.py
用vim编辑器新建一个FileStreaming.py代码文件，请在里面输入以下代码：
#!/usr/bin/env python3
from pyspark import SparkContext, SparkConf
from pyspark.streaming import
StreamingContext
conf = SparkConf()
conf.setAppName('TestDStream')
conf.setMaster('local[2]')
sc = SparkContext(conf = conf)
ssc =
StreamingContext(sc, 10)
lines = ssc.textFileStream('file:///usr/local/spark/mycode/streaming/logfile')
words = lines.flatMap(lambda line: line.split(' '))
wordCounts = words.map(lambda x : (x,1)).reduceByKey(lambda a,b:a+b)
wordCounts.pprint()
ssc.
start()
ssc.awaitTermination()
$ cd /usr/local/spark/mycode/streaming/logfile/
$ /usr/local/spark/bin/spark-submit FileStreaming.py
2.使用套接字流作为数据源
$ cd /usr/local/spark/mycode
$ mkdir streaming #如果已经存在该目录，则不用创建
$ cd streaming
$ mkdir socket
$ cd socket
$ vim NetworkWordCount.py
在NetworkWordCount.py文件中输入如下内容：
#!/usr/bin/env python3
from __future__ import print_function
import sys
from pyspark import SparkContext
from pyspark.streaming import
StreamingContext
if __name__ == "__main__":
if len(sys.argv) != 3:
print("Usage: NetworkWordCount.py <hostname> <port>", file=sys.stderr)
exit(-1)
sc = SparkContext(appName="PythonStreamingNetworkWordCount")
ssc =
StreamingContext(sc, 1)
lines = ssc.socketTextStream(sys.argv[1], int(sys.argv[2]))
counts = lines.flatMap(lambda line: line.split(" ")) \
.map(lambda word: (word, 1)) \
.reduceByKey(lambda a, b: a+b)
counts.pprint()
ssc.
start()
ssc.awaitTermination()
新打开一个窗口作为nc窗口，启动nc程序
$ nc -lk 9999
再新建一个终端（记作“流计算终端”），执行如下代码启动流计算：
$ cd /usr/local/spark/mycode/streaming/socket
$ /usr/local/spark/bin/spark-submit NetworkWordCount.py localhost 9999
可以在nc窗口中随意输入一些单词，监听窗口就会自动获得单词数据流信息，在监听窗口每隔1秒就会打印出词频统计信息，大概会在屏幕上出现类似如下的结果：
-------------------------------------------
Time: 2018-12-24 11:30:26
-------------------------------------------
('Spark', 1)
('love', 1)
('I', 1)
(spark,1) 下面我们再前进一步，把数据源头的产生方式修改一下，不要使用nc程序，而是采用自己编写的程序产生Socket数据源
$ cd /usr/local/spark/mycode/streaming/socket
$ vim DataSourceSocket.py
#!/usr/bin/env python3
import socket
# 生成socket对象
server = socket.socket()
# 绑定ip和端口
server.bind(('localhost', 9999))
# 监听绑定的端口
server.listen(1)
while 1:
# 为了方便识别，打印一个“我在等待”
print("I'm waiting the connect...")
# 这里用两个值接受，因为连接上之后使用的是客户端发来请求的这个实例
# 所以下面的传输要使用conn实例操作
conn,addr = server.accept()
# 打印连接成功
print("Connect success! Connection is from %s " % addr[0])
# 打印正在发送数据
print('Sending data...')
conn.send('I love hadoop I love spark hadoop is good spark is fast'.encode())
conn.close()
print('Connection is broken.‘)
执行如下命令启动Socket服务端$ cd /usr/local/spark/mycode/streaming/socket
$ /usr/local/spark/bin/spark-submit DataSourceSocket.py
启动客户端，即NetworkWordCount程序。新建一个终端（记作“流计算终端”），输入以下命令启动NetworkWordCount程序$ cd /usr/local/spark/mycode/streaming/socket
$ /usr/local/spark/bin/spark-submit NetworkWordCount.py localhost 9999
-------------------------------------------
Time: 2018-12-30 15:16:17
-------------------------------------------
('good', 1)
('hadoop', 2)
('is', 2)
('love', 2)
('spark', 2)
('I', 2)
('fast', 1)根据以上实现实验
本文介绍了StreamingContext在启动后无法直接停止的特点，并提供了一种临时停止实时流处理的方法：使用sc.stop(false)来暂停流处理任务而不真正关闭StreamingContext。
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
1044
