qq_44486439的博客

Python计算豆瓣热门电影分类TF-IDF环境PyCharm 2020.2.4 (Professional Edition)bs4==0.0.1urllib3==1.24.2MongoDB 3.4.10Python 3.7.4pymongo 3.11.3TF词频（Term Frequency, TF）反映语料库中的词条termiterm_itermi​在文档documentjdocument_jdocumentj​中出现的频率TFi,j=countitotal_countTF_{i

Spark连接Redis数据库环境spark-redis 2.5.0java 12.0.1Spark version 3.0.0-preview2Scala 2.12.10Apache Maven 3.6.3阿里云ECSCentOS Linux release 8.1.1911 (Core)Redis server v=6.0.9java version "12.0.1" 2019-04-16IntelliJ IDEA 2020.2.3 (Ultimate Edition)Mave

Spark连接MySQL数据库环境java 12.0.1Spark version 3.0.0-preview2Scala 2.12.10Apache Maven 3.6.3MySQL Server version: 5.7.18-20170830-log 20170531mysql-connector-java-8.0.21.jarIntelliJ IDEA 2020.2.3 (Ultimate Edition)Maven配置文件<?xml version="1.0" enco

Python Spark 读取多行JSON文件环境Python 3.7.6CentOS Linux release 8.1.1911 (Core) Spark version 3.0.0Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.2)实现代码from pyspark import SparkContext, SQLContextimport argparseif __name__ == "_

Python Spark实现协同过滤算法环境Python 3.7.6CentOS Linux release 8.1.1911 (Core) Spark version 3.0.0Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.2)实现代码from pyspark import SparkContext# $example on$from pyspark.mllib.recommendation i

Python Spark 实现逻辑回归算法环境Python 3.7.6CentOS Linux release 8.1.1911 (Core) Spark version 3.0.0Scala version 2.12.10 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.2)实现代码from pyspark import SparkContextfrom pyspark.mllib.regression import LabeledPoint

Java Spark读取JSON文件环境CentOS Linux release 8.1.1911 (Core)Spark version 3.0.0java 14.0.2Apache Maven 3.6.3目录结构./target目录由mvn编译打包生成.├── pom.xml├── src│ └── main│ └── java│ └── com│ └── data│ └─

Java构建Spark独立应用环境CentOS Linux release 8.1.1911 (Core)Spark version 3.0.0java 14.0.2目录结构./target目录由mvn编译打包生成.├── pom.xml├── src│ └── main│ └── java│ └── com│ └── demo│ └── spark│

CentOS环境中，在Python程序中创建Spark对象并进行行数统计

Scala实现文本词频统计环境CentOS Linux release 8.1.1911 (Core)Scala 2.13.3 (Java HotSpot(TM) 64-Bit Server VM, Java 14.0.2)实现代码import scala.io._object WordFrequenciesCounter { /* Description: 词频统计主函数 Args: args : main函数参数 Returns: 无 */ def