echo245的博客

1.maven项目pom.xml <dependencies> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifactId> <version>3.0.0</version><!--

部署spark任务在Kubernetes上报错使用官方案例部署失败`21/07/14 02:19:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicableUsing Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties

1，读数据库数据实例 def getDataForDb() : DataFrame ={ //获取sparkSession val spark: SparkSession = SparkSession.builder().master("local").appName("getData") .config("spark.sql.shuffle.partitions", 1).getOrCreate() val properties: Properties = n

1.报错scala.util.matching.Regex.(Ljava/lang/String;Lscala/collection/Seq;)VException in thread "main" java.lang.NoSuchMethodError: scala.util.matching.Regex.<init>(Ljava/lang/String;Lscala/collection/Seq;)V at scala.collection.immutable.StringLike

root@node01:/usr/lib/spark/spark-3.1.2-bin-hadoop3.2# sbin/start-workers.sh root@node02's password: root@node01's password: node01: Permission denied, please try again.

Spark在大多数情况下适合Scala连在一起的，因此我们在安装Spark之前不妨先安装Spark1.安装配置Scalatar -zxvf scala-2.13.5.tgz -C /usr/scala配置环境变量# 打开profile文件vi /etc/profile# 配置变量export SCALA_HOME= /usr/scala/scala-2.13.5export PATH=$PATH:$SCALA_HOME/bin让变量生效source /etc/profile

1.创建hadoop用户$ sudo useradd -m hadoop -s /bin/bash #创建hadoop用户，并使用/bin/bash作为shell $ sudo passwd hadoop #为hadoop用户设置密码，之后需要连续输入两次密码 $ sudo adduser hadoop sudo #为hadoop用户增加管理员权限 $ su - hadoop #切换

一、配置jdk环境1.在jdk官网下载jdk包2.使用ssh链接工具将所下载的jdk安装包拖进ubuntu虚拟机中/或者用u盘中转，该文章有相关操作https://blog.youkuaiyun.com/echo245/article/details/1152557683.打开终端控制，进入rootsudo su root4.创建Java的目标路径文件夹mkdir /usr/lib/jvm5.解压jdk包...

设置Ubuntu虚拟机中usb属性一般情况下新电脑的usb都是3.0就直接选3.1就好将文件复制到u盘再将u盘从主机文件夹中退出，然后重启虚拟机，重新插入u盘出现弹框确定即可

新建maven工程项目添加pom.xml依赖 <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core --> <dependency> <groupId>org.apache.spark</groupId> <artifactId>spark-core_2.12</artifact

遇到了一个小问题，就是无法读取文件控制台报错具体错误定位：Exception in thread "main" java.lang.RuntimeException: Error while running command to get file permissions : java.io.IOException: (null) entry in command string: null ls -F C:\Users\echo\IdeaProjects\spark-study\datas\test0