pyspark wordCount

最新推荐文章于 2022-05-11 21:02:41 发布

原创最新推荐文章于 2022-05-11 21:02:41 发布 · 211 阅读

0 ·

CC 4.0 BY-SA版权

技术博客专栏收录该内容

7 篇文章

订阅专栏

本文介绍如何利用Google Colab和PySpark进行文本文件的数据处理，包括安装必要的Java环境、配置环境变量、读取文本文件、去除标点符号、拆分单词、计数并展示出现频率最高的20个单词。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

from google.colab import drive
drive.mount(’/content/drive’)

ch=open(’/content/drive/My Drive/Colab Notebooks/title.txt’,‘r’)

import os #importing os to set environment variable
def install_java():
!apt-get install -y openjdk-8-jdk-headless -qq > /dev/null #install openjdk
os.environ[“JAVA_HOME”] = “/usr/lib/jvm/java-8-openjdk-amd64” #set environment variable
!java -version #check java version
install_java()

pip install pyspark

import pyspark
import re, os.path, time
from pyspark import SparkContext

sc = SparkContext()

def removePunctuation(text):
return re.sub(r’[^a-z0-9\s]’, ‘’, text.lower().strip())

shakespeareRDD = (sc.textFile(’/content/drive/My Drive/Colab Notebooks/title.txt’, 8).map(removePunctuation))

shakespeareWordsRDD = shakespeareRDD.flatMap(lambda line: line.split())

shakeWordsRDD = shakespeareWordsRDD.filter(lambda word: len(word) > 0)

def wordCount(shakeWordsRDD):
counts = shakeWordsRDD.map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
return counts

top20WordsAndCounts = wordCount(shakeWordsRDD).takeOrdered(20, key=lambda x: -x[1])
print(’\n’.join(map(lambda x: ‘{0}: {1}’.format(x[0], x[1]), top20WordsAndCounts)))