窦永厚

sparkstream定时清除日志和重启

vim /conf/spark-env.sh#!/usr/bin/env bashexport JAVA_HOME=/opt/module/jdk1.8.0_221export SCALA_HOME=/opt/module/scala-2.13.5export HADOOP_HOME=/opt/module/hadoop-3.1.4export HADOOP_CONF_DIR=/opt/module/hadoop-3.1.4/etc/hadoopexport SPARK_MASTER_IP=n

虚拟内存超出运行报错信息：Container [pid=30866,containerID=container_1600927953860_0003_02_000001] is running beyond virtual memory limits. Current usage: 117.3 MB of 1 GB physical memory used; 2.2 GB of 2.1 GB virtual memory used. Killing container.yarn Container

采用python开发spark sql简单入门1.读取本地csv文件转换为DataFrame2.DataFrame注册为spark sql临时表3.spark sql()函数查询返回DataFrame数据，或者直接DataFrame API返回数据

spark-submit提交python作业spark-submit执行pyspark脚本指定python环境提交：spark-submit --conf "spark.pyspark.driver.python=/usr/bin/python3.5" --conf "spark.pyspark.python=/usr/bin/python3.5" main.py

spark整合hive配置spark整合hive配置1.hive环境配置完成，略~~2.JAR包3.hive-site.xml4.测试spark整合hive配置1.hive环境配置完成，略~~2.JAR包cp {HIVE_HOME}/lib/mysql-connector-java-5.1.44-bin.jar {SPARK_HOME}/jars/cp {HIVE_HOME}/conf/hive-site.xml {SPARK_HOME}/conf3.hive-site.xml编辑{SPAR

ideaI快速入门一站式spark编程，下载安装idea、配置JDK3，maven，Scala，spark