AlphaTan1994-优快云博客

原创 TensorFlow Read TFRecords

Example Code:import tensorflow as tfrecord_path = '/tmp/tfrecord_write_test/part-r-00000'filename_queue = tf.train.string_input_producer([record_path])reader = tf.TFRecordReader()_, serialized_...

2019-10-20 20:15:56 188

原创 Read TFRecords get Exception Can't parse serialized Example

问题现象StacktraceInvalidArgumentError (see above for traceback): Key: features. Can't parse serialized Example. [[node ParseSingleExample/ParseSingleExample (defined at /work/tanyan/project/foresee/...

2019-10-20 19:58:16 702

原创 Spark任务偶现Task卡住很长时间导致Stage整体耗时长

问题现象提交大量Spark任务，概率性出现个别Task卡住一段时间，进而导致Stage整体耗时开销异常。问题分析采样Job836异常Stage2249 -> 卡住Task8：对应Executor日志：...INFO | [Executor task launch worker-78] | Running task 8.0 in stage 2249.0 (T...

2018-02-24 22:35:37 12636

原创 Spark访问安全HBase异常No valid credentials provided

问题现象在Spark（cluster模式）上访问安全HBase抛出如下异常：...DEBUG | Connecting to linux18/10.75.202.18:21310 | org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOStreams(RpcClientImpl.java:713)INFO | RPC ...

2018-02-24 21:38:06 9273 2

原创 Spark使用小窍门

Spark设置日志级别在Spark Driver实现中，初始化context实例之后，可以通过setLogLevel()方法设置日志级别。...JavaSparkContext context = new JavaSparkContext(conf);context.setLogLevel("DEBUG");......

2018-02-24 20:48:47 346

原创 Spark任务异常The auxService spaark_shuffle does not exist

问题现象Spark任务抛出如下异常：org.apache.hadoop.yarm.exception.InvalidAuxServiceException: The auxService:spark_shuffle does not exist问题解决方案于yarn-site.xml修改或添加如下配置：<property> <name>ya...

2018-02-24 20:41:26 3418 1

原创 SparkFiles.get NullPointerException

问题现象Spark任务抛出如下异常：Exception in thread "main" java.lang.NullPointerException at org.apache.spark.SparkFiles$.getRootDirectory(SparkFiles.scala:37) at org.apache.spark.SparkFiles$.get(SparkF...

2018-02-24 20:32:26 919

原创 Hive execute SQL抛出异常HiveSQLException: Invalid SessionHandle

问题现象Hive JDBC维持一个Connection长连接，在业务闲置一段时间后，执行SQL抛出如下异常：org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle [9ae14bf7-f921-4bf1-a616-0537132ce156] at org.apache.hive.jdbc.Utils.v

2018-02-05 20:37:49 14810

原创 Windows提交MapReduce异常，提示Shell output: main: command provided 1

问题现象Windows下提交MapReduce任务抛出如下异常：Diagnostics Exception from container-launch.Container id container_e06_1501570910088_0010_02_000001Exit code 1Stack trace ExitCodeException exitCode=1: at or

2018-01-20 09:39:55 2064 1

原创 HDFS Client UnknownHostException: NAMESERVICE

问题现象HDFS Client抛出如下异常：java.net.UnknownHostException: NAMESERVICE at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeProxi

2018-01-16 20:39:37 3902

原创 HDFS Client couldn't create proxy provider class

问题现象HDFS Client抛出如下异常：java.lang.RuntimeException: java.io.IOException: Couldn't create proxy provider class org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider at org.apa

2018-01-15 23:38:53 14776

原创 HDFS读取append中文件发生异常can't locate block

问题现象HDFS读取文件偶现失败，均为append中文件，抛出cant’t locate block异常。问题分析HDFS2.7.2及以前版本对Append操作处理存在实现缺陷。Append数据在写最后一个Block时，NameNode会刷新其Block Id，而DataNode会在写完后才刷新Block Id，在此期间如果业务侧同时读取该文件，并读到该Block，且其处在Data

2018-01-15 23:18:11 311

原创 HDFS Client偶现Filesystem closed

问题现象不定期偶现同一进程内所有业务访问HDFS抛出如下异常：java.io.IOException: Filesystem closed at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:703) at org.apache.hadoop.hdfs.DFSInputStream.readWithStr

2018-01-15 20:58:39 1286 1

原创 HDFS No Common protection layer between client and server

问题现象HDFS Client访问Server出现如下异常：...Caused by: javax.security.sasl.SaslException: No common protection layer between client and server at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHands

2018-01-15 20:10:29 7185 2

原创 SpringBoot基础工程搭建

写在前面前日里心血来潮，学习了一下Spring Boot框架。但是因为笔者智商有限，在浏览许多教程时，倍感吃力。于是写下本文，旨在快速的指导搭建一个最简易的Spring Boot Hello World工程，帮助那些像笔者一样不知从何下手的新人迈出第一步。构建Maven工程首先我们需要建立一个Maven工程（读者们可以选用自己喜欢的IDE，笔者采用的是IDEA）：许多有资深经验

2018-01-13 21:35:04 1256

原创 HBase on Spark NioWorkerPool VerifyError

问题现象提交HBase on Spark的Yarn Cluster任务时，AM启动失败，抛出如下异常：ERROR [2016-08-22 14:54:47,127] ({sparkDriverActorSystem-akka.actor.default-dispatcher-7} Slf4jLogger.scala[apply$mcV$sp]:66) - Uncaught fatal er

2018-01-13 16:45:06 331

原创 BulkLoad Can not create a path from a null string

问题现象高于1.0的HBase进行BulkLoad抛出异常：java.lang.IllegalArgumentException: Can not create a Path from a null string at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122) at org.apache.hadoop.fs.

2018-01-13 11:02:32 2808

原创一小时入手ANT脚本

写在前面最近因为项目愿意，笔者现学现用了一波ANT脚本。在这个过程中，愈加觉得它是一个很犀利的脚本工具，上手难度低，实现简单粗暴，功能也很强大，于是写了这样一篇文章，将其推荐给大家，希望能对大家有所帮助。ANT安装下载ANT包http://www-us.apache.org/dist//ant/binaries/apache-ant-1.9.7-bin.zip解压压缩包到任意目录配置环境变量例:

2018-01-13 10:19:41 375

独行的卡卡的博客