学习进步

查看报错日志：yarn logs applicationIDspark2-submit --master yarn --class com.hik.ReadHdfs test-1.0-SNAPSHOT.jar进入$SPARK_HOME目录，输入bin/spark-submit --help可以得到该命令的使用帮助。hadoop@wyy :/app/hadoop/spark100$ bin/spark-submit --helpUsage: spark-submit [options] <ap

1、启动hadoop所有进程start-all.sh等价于start-dfs.sh + start-yarn.sh但是一般不推荐使用start-all.sh(因为开源框架中内部命令启动有很多问题)。2、单进程启动。sbin/start-dfs.sh--------------- sbin/hadoop-daemons.sh --config .. --hostname .. start namenode ... sbin/hadoop-daemons.sh --conf...

hive -e "select * from ..." > /tmp/datahive -f 文件INSERT OVERWRITE TABLE employeesPARTITION (country = 'US', state = 'OR')SELECT * FROM staged_employees seWHERE se.cnty = 'US' AND se.st = 'OR';假如需要对多个分区执行，需要扫描表多次FROM staged_employees seINSERT .

val view = ss.sql( """ | select field1,concat_ws(",",collect_set(field2)) as convert_field | from t1 | where field1 is not null | group by field1 |""".stripMargin)

代码： val row2columnStr = """ |select * from temp_view |lateral view explode(split(field, ',')) tmpTable as result_view """.stripMargin

maven依赖<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/ma

在用spark进行数据处理过程中，避免不了行转列和列传行的操作，特此记录：1.列传行：这里举的例子是certificate_id ,telephone_number 每个身份证号可能对应多个手机号码df.createTempView("tmp") val result = sparkSession.sql( """ | select certifi...

windows环境本地起kafka producer进行测试，windows环境安装启动kafka可参照这篇博客：https://blog.youkuaiyun.com/shenyanwei/article/details/90374859代码如下：import org.apache.kafka.clients.consumer.ConsumerConfigimport org.apache.ka...

package sparkUtilimport org.apache.kudu.spark.kudu.KuduContextimport org.apache.spark.SparkConfimport org.apache.spark.sql.SparkSessionobject SparkKudu { //kuduMasters and tableName val kud...

计算结果数据需要写入Hbase，特此记录首先贴出hbase需要的maven依赖<hbase.version>1.2.3</hbase.version> <dependency> <groupId>org.apache.hbase</groupId> <artif...