<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[qq_37597249的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/qq_37597249</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; qq_37597249]]></copyright><item><title><![CDATA[spark on yarn 资源计算公式]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/104004219</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/104004219</guid><author>qq_37597249</author><pubDate>Thu, 16 Jan 2020 14:28:31 +0800</pubDate><description><![CDATA[一、spark on yarn 提交任务有两种模式

1、cluster模式 :

ApplicationMaster进程进行资源分配和executor的启动

提交命令：

--master yarn \

--deploy-mode client \

--driver-memory 5G \ applicationMaster 所在的容器分配的内存

--driver-cores 5 \ a...]]></description><category></category></item><item><title><![CDATA[Spark 2.X 读写 Oracel  代码实例      作者：刘大少]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/82349158</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/82349158</guid><author>qq_37597249</author><pubDate>Mon, 03 Sep 2018 15:29:08 +0800</pubDate><description><![CDATA[
import org.apache.spark.sql.{SaveMode, SparkSession}
import org.apache.spark.sql.jdbc.JdbcDialects
import org.apache.spark.sql.jdbc._
import org.apache.spark.sql.types._

一、读Oracle 表

// 参数自己传进来

1、定...]]></description><category></category></item><item><title><![CDATA[SPARK-UDF yyyy-MM-dd HH:mm:ss.SSS 类型转  秒+微秒的时间戳]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/81069079</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/81069079</guid><author>qq_37597249</author><pubDate>Mon, 16 Jul 2018 17:51:18 +0800</pubDate><description><![CDATA[
session.udf.register(&quot;toMMS&quot;,(str:String) =&amp;gt;{
  val sdf = new SimpleDateFormat(&quot;yyyy-MM-dd HH:mm:ss.SSS&quot;)
  val sdf2 = new SimpleDateFormat(&quot;yyyy-MM-dd HH:mm:ss&quot;)
  var st =&quot;&quot;
  if(str.length&amp;gt;1...]]></description><category></category></item><item><title><![CDATA[SPARK报错java.long.Exception……org.codehaus.commons.compiler.CompileException: File 'generated.java']]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/80774043</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/80774043</guid><author>qq_37597249</author><pubDate>Fri, 22 Jun 2018 15:10:00 +0800</pubDate><description><![CDATA[报错信息：

 java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 61, Column 30: Index expression of ...]]></description><category></category></item><item><title><![CDATA[SparkSQL报错:org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/80678068</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/80678068</guid><author>qq_37597249</author><pubDate>Wed, 13 Jun 2018 13:56:20 +0800</pubDate><description><![CDATA[错误信息：06/13/18 13:17:50  06/14/18 13:17:50  krbtgt/HADOOP.COM@HADOOP.COM18/06/13 13:17:59 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under S...]]></description><category></category></item><item><title><![CDATA[Container killed by YARN for exceeding memory limits. 15.6 GB of 15.5 GB physical memory used. Consi]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/80612835</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/80612835</guid><author>qq_37597249</author><pubDate>Thu, 07 Jun 2018 17:42:45 +0800</pubDate><description><![CDATA[spark 报错
    [Stage 45:(482 + 46) / 528][Stage 46:(64 + 148) / 528][Stage 55:&amp;gt;(27 + 0) / 528]18/06/07 16:07:11 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Container killed by YARN for exceedin...]]></description><category></category></item><item><title><![CDATA[关闭 centos7 的滴滴声]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/80535345</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/80535345</guid><author>qq_37597249</author><pubDate>Fri, 01 Jun 2018 11:22:06 +0800</pubDate><description><![CDATA[1.执行：vi /etc/inputrc    #set bell-style none 把这一行的#去掉2、执行：vi ~/.bashrc   添加：setterm –blength 03、执行：reboot 声音关闭]]></description><category></category></item><item><title><![CDATA[spark 给hive表填加自增字段和UUID]]></title><link>https://blog.youkuaiyun.com/qq_37597249/article/details/80534257</link><guid>https://blog.youkuaiyun.com/qq_37597249/article/details/80534257</guid><author>qq_37597249</author><pubDate>Fri, 01 Jun 2018 10:32:29 +0800</pubDate><description><![CDATA[直接操作表的时候用自增这个方式会报错

因为spark over里必须有字段  在hive中使用
    直接用SQL : create table table2Name as select row_number() over () as row_id,* from table1Name
UUID主键：也可以操作表 也可以操作RDD 可以同时 这种在增加UUID之后必须cache 他会一直到触发a...]]></description><category></category></item></channel></rss>