<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[xwq的博客]]></title><description><![CDATA[重在分享，互相学习。]]></description><link>https://blog.youkuaiyun.com/MileyPriencess</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; MileyPriencess]]></copyright><item><title><![CDATA[org.apache.spark.SparkException: Task not serializable]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/99694263</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/99694263</guid><author>MileyPriencess</author><pubDate>Sat, 17 Aug 2019 16:10:33 +0800</pubDate><description><![CDATA[在spark shell中运行下述代码：
    val max_array = max_read_fav_share_vote.collect
    val max_read = max_array(0)(0).toString.toDouble
    val max_fav = max_array(0)(1).toString.toDouble
    val max_share = ma...]]></description><category></category></item><item><title><![CDATA[java.lang.OutOfMemoryError: GC overhead limit exceeded]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/98728707</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/98728707</guid><author>MileyPriencess</author><pubDate>Wed, 07 Aug 2019 10:59:40 +0800</pubDate><description><![CDATA[在spark1.6.0中运行代码，在log中出现如下错误提示：
java.lang.OutOfMemoryError: GC overhead limit exceeded

这个一般是因为在垃圾回收中占用了大量时间造成的。通常来说，当程序用98%的时间回收了不到2%的堆内存时导致的。通常是设置的堆内存太小，导致没有足够的内存。
1.spark.executor.memory 调大参数扩大内存
2...]]></description><category></category></item><item><title><![CDATA[ERROR util.Utils: uncaught error in thread SparkListenerBus, stopping SparkContext]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/98727999</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/98727999</guid><author>MileyPriencess</author><pubDate>Wed, 07 Aug 2019 10:05:42 +0800</pubDate><description><![CDATA[在spark1.6.0中运行代码，在log中出现如下错误提示：
ERROR util.Utils: uncaught error in thread SparkListenerBus, stopping SparkContext

这个错误提示可以忽略不记。
当我们清理并终止守护进程上下文清理线程时，在执行结束时会打印这个。今后这个错误提示可能会消除，因为它可能会让用户感到困惑。
https://...]]></description><category></category></item><item><title><![CDATA[org.apache.spark.SparkException: Failed to get broadcast_270_piece0 of broadcast_270]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/98514212</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/98514212</guid><author>MileyPriencess</author><pubDate>Mon, 05 Aug 2019 20:22:12 +0800</pubDate><description><![CDATA[在spark1.6.0中运行代码，出现如下错误：
org.apache.spark.SparkException: Failed to get broadcast_270_piece0 of broadcast_270

解决方法
1.可能是因为spark.cleaner.ttl导致的，spark.cleaner.ttl设置一个清除时间，使spark清除超过这个时间的所有RDD数据，以便腾出空间给...]]></description><category></category></item><item><title><![CDATA[Shell脚本]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/95966558</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/95966558</guid><author>MileyPriencess</author><pubDate>Mon, 15 Jul 2019 14:52:00 +0800</pubDate><description><![CDATA[#! 告诉系统其后路径所指定的程序即是解释此脚本文件的 Shell 程序。
在Linux中，直接用/bin/sh test.sh指令运行shell脚本文件，这里是调用Bourne Shell解释器。
1. Shell变量
定义变量时，变量名不加美元符号（$）变量名和等号之间不能有空格。同时，变量名的命名须遵循如下规则：
命名只能使用英文字母，数字和下划线，首个字符不能以数字开头。
中间不能有空格，...]]></description><category></category></item><item><title><![CDATA[Spark Shell -- <console>:29: error: object y is not a member of package x]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/95229441</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/95229441</guid><author>MileyPriencess</author><pubDate>Tue, 09 Jul 2019 19:56:20 +0800</pubDate><description><![CDATA[在spark shell中导入一个包出错：
import x.y

但是报错：
 &lt;console&gt;:29: error: object y is not a member of package x

解决方法：
打开spark shell时将x.y包加入
bin/spark-shell --packages x.y

也看到有人说加上 root 有效：
import  _root_....]]></description><category></category></item><item><title><![CDATA[Error: Must specify a primary resource (JAR or Python or R file)  --spark--scala]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/94430109</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/94430109</guid><author>MileyPriencess</author><pubDate>Tue, 02 Jul 2019 09:30:28 +0800</pubDate><description><![CDATA[在spark集群上运行一个shell脚本，出现错误：
出现这个问题的原因是shell脚本中没有指定主资源，这是因为我在shell脚本中有多个jar包，但都用来作为依赖资源而没有指定主资源。
类似运行如下的指令会出现这个错误：
./bin/spark-submit --class Sentimenter --master local[4] --jars /home/ubuntu/spark/spar...]]></description><category></category></item><item><title><![CDATA[Intellij IDEA中编写Scala程序报错：Error:scalac: 'jvm-1.8' is not a valid choice for '-target']]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/91619625</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/91619625</guid><author>MileyPriencess</author><pubDate>Wed, 12 Jun 2019 20:53:10 +0800</pubDate><description><![CDATA[在Intellij IDEA中编写Scala程序报错：Error:scalac: ‘jvm-1.8’ is not a valid choice for ‘-target’
Intellij IDEA版本是2019.1.3，
JDK版本是1.8.0，
Scala版本是2.10.5.
解决方法：根据以下路径File-&gt;Settings -&gt; Build, Execution, Deplo...]]></description><category></category></item><item><title><![CDATA[基于WEKA的线性回归代码实现（最小二乘法）]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/89067694</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/89067694</guid><author>MileyPriencess</author><pubDate>Sun, 07 Apr 2019 12:19:57 +0800</pubDate><description><![CDATA[1.最小二乘法求线性回归参数
线性回归公式

损失函数

最小二乘法求参数
令
，，
因此，损失函数转换为

求导求出驻点得到

2.代码
package weka.classifiers.myAlgorithm;

import weka.classifiers.Classifier;
import weka.core.Instance;
import weka.core.Instances;
...]]></description><category></category></item><item><title><![CDATA[基于WEKA的朴素贝叶斯（Naive Bayes）代码实现]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/89066324</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/89066324</guid><author>MileyPriencess</author><pubDate>Sun, 07 Apr 2019 10:42:21 +0800</pubDate><description><![CDATA[1.Naive Bayes公式


2.代码
package weka.classifiers.myAlgorithm;

import weka.classifiers.Classifier;
import weka.core.Instance;
import weka.core.Instances;

public class NB_xu extends Classifier
{
	
	pub...]]></description><category></category></item><item><title><![CDATA[基于WEKA的K近邻（KNN）代码实现]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886702</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886702</guid><author>MileyPriencess</author><pubDate>Fri, 29 Mar 2019 11:04:22 +0800</pubDate><description><![CDATA[package weka.classifiers.xwq;

import weka.classifiers.Classifier;
import weka.core.Instance;
import weka.core.Instances;

public class KNN_xu extends Classifier
{
	/**
	 * 训练集
	 */
	public Instances ...]]></description><category></category></item><item><title><![CDATA[基于WEKA的决策树（ID3）代码实现]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886615</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886615</guid><author>MileyPriencess</author><pubDate>Fri, 29 Mar 2019 11:01:30 +0800</pubDate><description><![CDATA[package weka.classifiers.xwq;

import java.util.ArrayList;
import weka.classifiers.Classifier;
import weka.classifiers.Evaluation;
import weka.core.Attribute;
import weka.core.Instance;
import weka.co...]]></description><category></category></item><item><title><![CDATA[Eclipse  java.io.NotSerializableException]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886118</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/88886118</guid><author>MileyPriencess</author><pubDate>Fri, 29 Mar 2019 10:44:09 +0800</pubDate><description><![CDATA[在Eclipse的一个类文件中编写另一个类，运行时出现java.io.NotSerializableException错误提示。具体如下：
java.io.NotSerializableException: weka.classifiers.teacher.ID3_xu$TreeNodes
	java.io.ObjectOutputStream.writeObject0(Unknown Sourc...]]></description><category></category></item><item><title><![CDATA[最简单的导入weka源码到eclipse中的方法]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/84258111</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/84258111</guid><author>MileyPriencess</author><pubDate>Mon, 19 Nov 2018 18:52:24 +0800</pubDate><description><![CDATA[原先，是按照 https://blog.youkuaiyun.com/MileyPriencess/article/details/74315577 将weka源码导入到eclipse中的，相比于而言下面的方法是非常简单的了。
1.从weka官网下载压缩包，将压缩包解压后再将里面的weka-src压缩包解压，
2.在eclipse中右击新建java project，接下来如下图操作即可完成



如此即完成w...]]></description><category></category></item><item><title><![CDATA[SparseInstance和Instance的相互转换-稀疏矩阵与矩阵的相互转换]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83795573</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83795573</guid><author>MileyPriencess</author><pubDate>Tue, 06 Nov 2018 23:23:44 +0800</pubDate><description><![CDATA[当一个数据集中很多instance中的很多属性值是0时，这个数据集就会以稀疏矩阵的形式存储。这个过程主要是通过将Instance转换为SparseInstance实现的，SparseInstance是Instance的子类。
对于一个值为{0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,2,0,0}的instance，其对应的SparseInstance为{9 1, 18 ...]]></description><category></category></item><item><title><![CDATA[将文本数据转换成arff文件]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83795708</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83795708</guid><author>MileyPriencess</author><pubDate>Tue, 06 Nov 2018 23:08:10 +0800</pubDate><description><![CDATA[目的：
将类似下图的文本数据转换为arff文件。文本数据每一行是一个句子和该句子的类属性值（classValue），得到的arff文件中的一个实例（Instance）即为一个句子的单词向量和类属性值。

思路：
1.将原始文本数据中一些特殊符号去除，以免在构建单词向量时出现问题；
2.在去除特殊符号的文本数据上自动添加arff文件的表头，另存为arff文件；
3.在WEKA中读取该arff文件，再...]]></description><category></category></item><item><title><![CDATA[使用Maven导入LibRec到工程出现错误Could not calculate build plan: Plugin org.apache.maven.plugins:maven-resource]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83585613</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83585613</guid><author>MileyPriencess</author><pubDate>Wed, 31 Oct 2018 21:44:58 +0800</pubDate><description><![CDATA[在使用Maven将LibRec2.0.0文件导入到Eclipse工程中时，出现下面的问题：

出现这种问题的原因是Maven无法下载所需要的依赖库文件。
第一次出现这个问题，采用下述方法解决：
1）右键对应工程项目，进入Maven-&amp;gt;Update Project；
2)  选择Update dependencies 并勾上其下面的 Force Update of Snapshots/Rele...]]></description><category></category></item><item><title><![CDATA[ValueError: Floating point image RGB values must be in the 0..1 range.]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83341438</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83341438</guid><author>MileyPriencess</author><pubDate>Wed, 24 Oct 2018 11:31:38 +0800</pubDate><description><![CDATA[值错误：浮点图像RGB值必须在0…1范围内。
错误来源：在使用图像处理函数处理一个图片后，通过pyplot输出可视化时，出现上述错误提示。
错误原因：如错误提示所说，这是输出图像中像素的实数值超出0-1范围导致的。
解决方法：在输出最终图像之前需要将其值截断在0-1之间，具体操作如下：
after_img = tf.clip_by_value(before_img, 0.0, 1.0)

若不进行...]]></description><category></category></item><item><title><![CDATA[python之os.walk()]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83314342</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83314342</guid><author>MileyPriencess</author><pubDate>Tue, 23 Oct 2018 20:28:42 +0800</pubDate><description><![CDATA[功能：
os.walk() 是一个简单易用的文件、目录遍历器，可以帮助我们高效的处理文件、目录方面的事情。
声明：
walk(top, topdown=True, onerror=None, followlinks=False)
输入参数：

top：所需遍历目录的路径
topdown：True时，则优先遍历top目录，否则优先遍历top的子目录(默认为True)
onerror：需要一个 cal...]]></description><category></category></item><item><title><![CDATA[ValueError: Parent directory of /path/to/model/model.ckpt doesn't exist, can't save.]]></title><link>https://blog.youkuaiyun.com/MileyPriencess/article/details/83096626</link><guid>https://blog.youkuaiyun.com/MileyPriencess/article/details/83096626</guid><author>MileyPriencess</author><pubDate>Tue, 16 Oct 2018 20:07:52 +0800</pubDate><description><![CDATA[代码如下：
MODEL_SAVE_PATH = &quot;/path/to/mode/&quot;
MODEL_NAME = &quot;model.ckpt&quot;
saver.save( sess, os.path.join(MODEL_SAVE_PATH, MODEL_NAME),
			global_step=global_step )

即在使用tensorflow保存session时出现错误：
ValueError: ...]]></description><category></category></item></channel></rss>