E盘根目录创建test.txt输入测试内容如下:
this is a test
this very good
you is very good
what are you
完整代码如下:
from pyspark import SparkContext,SparkConf
def wordcount():
txtfile=r'E:\test.txt'
conf=SparkConf()
conf.setAppName("worlcount")
conf.setMaster("local[*]")
sc = SparkContext(conf=conf)
rdd=sc.textFile(txtfile)
rdd.flatMap(lambda x:x.split( )).map(lambda x:(x,1)).reduceByKey(lambda x,y:x+y).foreach(lambda x:print(x))
if __name__ == '__main__':
wordcount()
pycharm直接运行,或者使用命令提交:spark-submit --master local E:\geotrellis\pyspark_dem\base\wordcount.py
执行结果:
说明执行成功。