<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[DayOneMore的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/DayOneMore</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; DayOneMore]]></copyright><item><title><![CDATA[latex 枚举编号样式]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115592850</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115592850</guid><author>DayOneMore</author><pubDate>Sun, 11 Apr 2021 11:25:53 +0800</pubDate><description><![CDATA[\begin{enumerate}
\item [(1)]
\item [(2)]
\end{enumerate}

]]></description><category></category></item><item><title><![CDATA[latex 中手写l的输入方法]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115592574</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115592574</guid><author>DayOneMore</author><pubDate>Sun, 11 Apr 2021 11:19:27 +0800</pubDate><description><![CDATA[\partial 后面的 ℓ\ellℓ 的输入方法
\ell
ℓ\ellℓ

]]></description><category></category></item><item><title><![CDATA[Mac python2 python3共存]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115176018</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115176018</guid><author>DayOneMore</author><pubDate>Wed, 24 Mar 2021 14:16:47 +0800</pubDate><description><![CDATA[https://www.jianshu.com/p/d19a37e1bbd9
python3画图和python2不同，更好看一些

]]></description><category></category></item><item><title><![CDATA[python 常用画图总结]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115175825</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115175825</guid><author>DayOneMore</author><pubDate>Wed, 24 Mar 2021 14:15:21 +0800</pubDate><description><![CDATA[官方文档：https://matplotlib.org/stable/gallery/index.html
# coding=UTF-8
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
# import sys
# reload(sys)
# sys.setdefaultencoding('utf-8')


labels = ['5ms','10ms','50ms','100ms','500ms','1000ms',]]></description><category></category></item><item><title><![CDATA[ValueError matplotlib display text must have all code points 128 or use Unicode strings]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115141791</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115141791</guid><author>DayOneMore</author><pubDate>Tue, 23 Mar 2021 20:23:14 +0800</pubDate><description><![CDATA[错误问题：ValueError: matplotlib display text must have all code points &lt; 128 or use Unicode strings
错误原因：在使用matplotlib进行图表展示时，如果需要在图表上添加中文，可能会报字符类型的值错误。
解决办法：重新调用系统，修改默认的字符类型。
import sys
reload(sys)
sys.setdefaultencoding('utf-8')

说明：一般在我们写程序的过程中，经常会遇到这种字符]]></description><category></category></item><item><title><![CDATA[python中有中文报错]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/115141531</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/115141531</guid><author>DayOneMore</author><pubDate>Tue, 23 Mar 2021 20:20:53 +0800</pubDate><description><![CDATA[在python程序中，只要有中文就会报错，不管是否在注释里。
SyntaxError: Non-ASCII character '\xe5' in file paint.py on line 14, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details
原因
python的默认编码文件是用的ASCII码，python文件中使用了中文等非英语字符。
解决办法
在Python源文件的最开始一行，加入一句：
#]]></description><category></category></item><item><title><![CDATA[spark集群搭建]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114984370</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114984370</guid><author>DayOneMore</author><pubDate>Thu, 18 Mar 2021 17:25:18 +0800</pubDate><description><![CDATA[/etc/hosts中去掉127.0.1.1映射
个台机器之间互相ssh，并测试
安装java,可能会遇到这个问题，看

安装jdk，即使配置好环境变量，依旧报错


克隆spark，并解压
配置spark-env.sh
运行pyspark测试

安装scipy



...]]></description><category></category></item><item><title><![CDATA[ERROR TaskSchedulerImpl: Lost executor 1 on 1xx.xx.xxx.x: Remote RPC client disassociated. Likely du]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114882871</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114882871</guid><author>DayOneMore</author><pubDate>Tue, 16 Mar 2021 15:13:31 +0800</pubDate><description><![CDATA[ERROR TaskSchedulerImpl: Lost executor 1 on xx.xx.xx.xx: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
是SPARK_DRIVER_MEMORY或SPARK_EXECUTOR_MEMORY不足，在spark-env.sh中增大一些]]></description><category></category></item><item><title><![CDATA[spark SLF4J: Class path contains multiple SLF4J bindings.]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114819317</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114819317</guid><author>DayOneMore</author><pubDate>Mon, 15 Mar 2021 09:44:06 +0800</pubDate><description><![CDATA[spark 中出现这个问题
在spark-env.sh中，之前因为尝试解决别的问题，多加了个
export SPARK_DIST_CLASSPATH=$(/home/zhangjiahua/spark/hadoop-2.6.4 classpath)
删掉就行

]]></description><category></category></item><item><title><![CDATA[spark中failed to launch: nice -n]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114809848</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114809848</guid><author>DayOneMore</author><pubDate>Sun, 14 Mar 2021 23:38:34 +0800</pubDate><description><![CDATA[问题：
failed to launch: nice -n 0 /home/xxx/spark/spark-3.0.2-bin-hadoop2.7/bin/spark-class org.apache.spark.deploy.master.Master --host 1xx.1x.xx.xx --port 7077 --webui-port 8080

反复检查spark-env.sh都没有错，最后发现是spark-defalut.conf中
spark.network.timeout设置的太小，之前是2]]></description><category></category></item><item><title><![CDATA[gaierror: [Errno -2] Name or service not known]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114693117</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114693117</guid><author>DayOneMore</author><pubDate>Fri, 12 Mar 2021 12:47:21 +0800</pubDate><description><![CDATA[配置spark，运行的时候报错
/etc/hosts中，127.0.0.1的映射保留，127.0.1.1的要删掉

]]></description><category></category></item><item><title><![CDATA[Invalid maximum head size: -Xmx8192M]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114583820</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114583820</guid><author>DayOneMore</author><pubDate>Tue, 09 Mar 2021 15:16:06 +0800</pubDate><description><![CDATA[set("spark.executor.memory", "8g")如果设置的太大，会报下面的错误，减小就可以了


]]></description><category></category></item><item><title><![CDATA[Spark性能优化指南]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114581645</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114581645</guid><author>DayOneMore</author><pubDate>Tue, 09 Mar 2021 14:08:11 +0800</pubDate><description><![CDATA[非常好的两篇文章
Spark性能优化指南——基础篇
Spark性能优化指南——高级篇

]]></description><category></category></item><item><title><![CDATA[奇怪问题]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114329515</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114329515</guid><author>DayOneMore</author><pubDate>Sat, 06 Mar 2021 22:16:48 +0800</pubDate><description><![CDATA[奇怪的问题
前提：迭代100000的时候 步长100
问题描述

单机单线程运行，约13.5s
使用local[2]，local[3]，local[4]，local[5]模式运行，约5.9s运行完程序，但是会提示如下WARN

local[6]及以后模式下不会出现如上WARN，但是运行速度降低，大概要10s运行完程序，线程数越大好像越慢一些
3个worker（包括一台机器既作为master又作为worker），运行更慢，大概要24s
2个worker，大概要20s
1个worker大概要17s
本机既作为w]]></description><category></category></item><item><title><![CDATA[ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114360336</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114360336</guid><author>DayOneMore</author><pubDate>Sat, 06 Mar 2021 16:54:13 +0800</pubDate><description><![CDATA[问题描述

spark分布式运行的过程中，自己有两个worker，但是运行的时候，只有一台worker正常运行，另一台报了右边的错，如下图
ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM

把正常的机器停掉，只运行有问题的worker（dl21,dl23）,发现能出结果，但是两台机器的stderr仍然是右边的错，所以奇怪的是，结果怎么出来的？
网上说是因为资源分配不够，但是我分配了executor 10G资源，反而出现下面这种问题

然后有些]]></description><category></category></item><item><title><![CDATA[从本地上传文件到hdfs，并在程序中读区]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114337409</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114337409</guid><author>DayOneMore</author><pubDate>Wed, 03 Mar 2021 22:39:00 +0800</pubDate><description><![CDATA[背景
自己在使用spark MLlib的时候，在单机下反而比分布式下更快，所以猜测是从本地读数据比较慢，所以想把数据从本地弄到hdfs上
上传数据到hdfs

查看hdfs 根目录/下文件，发现有两种
hadoop fs -ls /
hdfs dfs -ls /
创建一个新的文件夹/data
Hadoop fs -mkdir /data
上传文件到hdfs
hadoop fs -put ~/project/test.txt /data
删除文件
hadoop fs -rm /data/xxx


...]]></description><category></category></item><item><title><![CDATA[spark-submit后发现程序没有分布式运行]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114169661</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114169661</guid><author>DayOneMore</author><pubDate>Wed, 03 Mar 2021 20:23:04 +0800</pubDate><description><![CDATA[问题描述：
spark-submit后发现程序没有分布式运行
w3m http://localhost:8080


发现要用--master spark://[masterip]:7077命令运行,例如
spark-submit --master spark://masternode:7077 yourapp
但是仍然不行，出现如下状况

8080 UI显示如下：


原因分析：
master和worker之间通信出了问题，因为用一台机器座位worker和master是可以运行的
仔细看，下图这里出了问题]]></description><category></category></item><item><title><![CDATA[spark中ImportError: No module named numpy原因和解决方法]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114326295</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114326295</guid><author>DayOneMore</author><pubDate>Wed, 03 Mar 2021 20:06:08 +0800</pubDate><description><![CDATA[问题描述
spark 集群运行的时候，报错
ImportError: No module named numpy
但是想来想去记得numpy都安装了，但其实不是的
问题分析
spark集群运行，需要所有的机器都有numpy,自己使用了一台新的机器dl21，这个机器原本没有自带numpy,需要安装
解决方案
sudo apt-get update
sudo apt-get install python-numpy python-scipy python-matplotlib ipython ipython-n]]></description><category></category></item><item><title><![CDATA[Linux配置了环境变量JAVA仍然是openjdk]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114321074</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114321074</guid><author>DayOneMore</author><pubDate>Wed, 03 Mar 2021 16:13:41 +0800</pubDate><description><![CDATA[问题描述
不使用openjdk，使用jdk,下载完后，也配置了环境变量
反复检查
/etc/profile
~/.bashrc

均没有错误，但是java -version仍旧是openjdk
解决方案
需要删除/usr/bin下的java文件

]]></description><category></category></item><item><title><![CDATA[Java HotSpot(TM) Server VM warning: You have loaded library /home/xxx/spark/hadoop-2.6.4/lib/native/]]></title><link>https://blog.youkuaiyun.com/DayOneMore/article/details/114287578</link><guid>https://blog.youkuaiyun.com/DayOneMore/article/details/114287578</guid><author>DayOneMore</author><pubDate>Wed, 03 Mar 2021 10:26:33 +0800</pubDate><description><![CDATA[问题描述：
Java HotSpot™ Server VM warning: You have loaded library /home/xxx/spark/hadoop-2.6.4/lib/native/libhadoop.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It’s highly recommended that you fix the library with 'ex]]></description><category></category></item></channel></rss>