Hadoop-0.20.2的log存放时间控制

最新推荐文章于 2022-05-10 17:01:53 发布
最新推荐文章于 2022-05-10 17:01:53 发布 · 275 阅读 · 0
文章标签：

#Hadoop

本文详细介绍了如何通过调整Hadoop配置文件，优化JobTracker的WebUI显示多个job执行状态并延长其存放时间，以及延长Tasklog在HistoryUI上的存放时间，以提升整体性能。

1. 想要在JobTracker的WebUI上（master:50030）多保持几个job执行状态，默认是100个

调整Hadoop配置文件，比如调整为200个

<property>

<name>mapred.jobtracker.completeuserjobs.maximum</name>

<value>200</value>

        </property>

 

 2. 想要在JobTracker的WebUI上延长job执行状态的存放时间，默认是24小时，现在调整为72小时，也就是259200000微秒

<property>

<name>mapred.jobtracker.retirejob.interval</name>

<value>259200000</value>

</property>

 

 3. 想要在JobTracker的HistoryUI上延长Task log（Last 4KB，Last 8KB， ALL）的存放时间，默认是24小时，这个参数是MapReduce job控制的，要么在Hadoop的mapred-site.xml中设置该值为final或者在每个MapReduce job中设置为72

<property>

<name>mapred.userlog.retain.hours</name>

<value>72</value>

</property>



iteye_19448
关注
Hadoop日志存放位置
03-11
Hadoop日志存放位置，存储规则和存储的位置，Hadoop日志存储修改等
hadoop配置
05-09
export HADOOP_LOG_DIR=${HADOOP_HOME}/logs # source /etc/profile ``` 6. **配置Hadoop环境**： - 配置Hadoop环境变量： ```bash # vi /home/hadoop/hadoopinstall/hadoop-0.20.2/conf/hadoop-env.sh ...
hadoop 2.4.1日志聚合设置，并设置自动回收时间
zhoujj303030的专栏
12-04 2420
1.hadoop 日志默认生产路径是各个datanode节点上 hadoop环境变量对应的logs文件夹下面： 设置完日志聚合之后生成的路径： 但是怎么样设置日志聚合呢？ yarn.log-aggregation-enable true yarn.lo
Hadoop日志存放问题
cloud-2014的专栏
04-11 451
Hadoop日志存放路径详解
hadoop日志文件_奇怪的时间
笨字第1号
07-17 2708
在hadoop性能优化的实验中，碰到一个奇怪的问题，描述如下： 在shell端显示                  job先执行时间：13：53：32～13：57：18（3m46s，根据shell终端显示计算）                  Map执行时间：13：53：33～13：57：04（3m31s，根据shell终端显示的估计值）                  Reduc执
Hadoop(5) 历史服务其和日志聚集的配置、集群时间同步
Alaskyed的博客
12-25 367
Hadoop(五) 历史服务其和日志聚集的配置、集群时间同步的配置 历史服务器 为了查看程序的历史运行情况,需要配置一下历史服务器 配置mapred-site.xml 打开mapred-site.xml,在configuration标签里面添加以下内容: <!-- 历史服务器端地址 --> <property> <name>mapreduce.jobhi...
Hadoop伪分布式环境搭建（hadoop-0.20.2、hive-0.11.0、pig-0.5.0、zookeeper-3.4.3）
cyw1900的专栏
05-08 1069
一．安装虚拟机及前期准备 1.      VMWare 下安装CentOS 6.0系统，网络制式采用NAT 2.      将hadoop添加到sudoers su  root 输入root的口令，成功后就换成了root用户 继续输入命令： chmod u+w/etc/sudoers vi/etc/sudoers 在此行：root ALL=(ALL:ALL) ALL 后加一行：
hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.3.3.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.3.3.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Null.java tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Null.java: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.8.3.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.8.3.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.3.5.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.3.5.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.8.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.8.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.3.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.3.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.22.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.22.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.9.1.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.9.1.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.1.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.1.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.20.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.20.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.0-alpha4.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.0-alpha4.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.9.2.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.9.2.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.0-alpha2.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.0-alpha2.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.2.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.2.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.10.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_2.10.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.1.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.0.1.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.1.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.1.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.4.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.2.4.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.21.0.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/hadoop-hdfs_0.21.0.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.3.xml tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/jdiff/Apache_Hadoop_HDFS_3.1.3.xml: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6-tests.jar tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-client-3.3.6-tests.jar: Cannot open: No such file or directory hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.6.jar tar: hadoop-3.3.6: Cannot mkdir: Permission denied tar: hadoop-3.3.6/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.3.6.jar: Cannot open: No such file or directory tar: Exiting with failure status due to previous errors jtx@ubuntu:~$
最新发布
09-28
chown -R hadoop:hadoop hadoop-0.20.2/ ``` 2. **使用 sudo 解压** 若需解压到系统目录（如 `/usr/local`），需提升权限： ```bash sudo tar -xzvf hadoop-3.3.6.tar.gz -C /usr/local ``` 3. **修改目录...
Hadoop详细解析
woaini886353的博客
05-10 3422
背景知识 1.1、企业数据部的一般组织结构 企业数据部的一般组织结构，适用于大中型企业。 1.2、企业数据部的业务流程分析 业务流程： 电商业务人员：针对活动专题页(活动的效果)有业务需求 活动页的用户访问数、用户下单数、用户支付数、用户退单数 数据部部门： 数据分析人员（写sql），结果对外提供 ...
hadoop伪分布式需修改的配置文件（5个）
weixin_34247032的博客
01-25 3884
2019独角兽企业重金招聘Python工程师标准>>> ...
hadoop提交任务后日志存放问题
ls386239766的专栏
08-21 790
hadoop提交任务后，会有日志记录此次执行的情况。目前这些日志都保留在hdfs文件系统中，生成了大量的小文件，从而耗费了大量的磁盘空间。如果每天提交的任务多且频繁，磁盘空间消耗的速率是比较惊人的。 可清理的日志路径： /user/history  /tmp   /user/root/.Trash 如下图所示，上面的部分是清除日志后磁盘的使用率与block块的数量，下图是没清理的。
Logrotate日志存储
枫叶斜阳的博客
07-17 465
Logrotate日志存储Logrotate日志存储类似格式如下 Logrotate日志存储 logrotate 程序是一个日志文件管理工具。用来把旧的日志文件删除，并创建新的日志文件，称为日志转储或滚动。可以根据日志文件的大小，也可以根据其天数来转储，这个过程一般通过 cron 程序来执行 配置文件是 /etc/logrotate.conf 主要参数如下 compress 通过gzip 压缩转储以后的日志 nocompress 不需要压缩时，用这个参数 copytruncate 用于还在打开中的日志文件，
hadoop配置日志的聚集
weixin_47260194的博客
03-03 631
hadoop创建日志聚集
Hadoop之配置日志的聚集
a2957827124的博客
04-07 1763
Hadoop之配置日志的聚集
Hadoop 历史服务器与日志详解
热门推荐
Zephyr's Blog
01-01 1万+
本文转至 Hadoop日志存放路径详解 一 Hadoop日志存放路径详解 1 Hadoop系统服务输出的日志 2 配置 Hadoop 系统服务日志 二 Mapreduce 程序相关的日志 1 历史作业日志 2 Container 日志 一. Hadoop日志存放路径详解 　Hadoop 的日志有很多种，很多初学者往往遇到错而不知道怎么办，其实这时候就应该去看看日志里面
Hadoop日志详解
逝无痕——kaidy
03-31 6743
文章目录Hadoop日志详解1 Hadoop 系统服务输出的日志1.1 修改Hadoop日志目录1.2 修改YARN日志目录2 Mapreduce程序输出来的日志2.1 作业运行日志2.2 任务运行日志 (Container 日志) Hadoop日志详解 Hadoop的日志大致可以分为两大类: 1、 Hadoop 系统服务输出的日志； 2、 Mapreduce程序输出来的日志。 且这两类的日志存放...
3、学大数据笔记-hadoop2的配置
weixin_30663471的博客
07-09 127
前面和3.0一样,创建用户,免密登录,从解压,安装开始 1、上传hadoop安装包,进行解压 #1.创建hadoop安装目录 [root@node1 ~]# mkdir -p /opt/bigdata #2.解压hadoop-2.7.3.tar.gz [root@node1 ~]# tar -xzvf hadoop-2.7.3.tar.gz -C /opt/bigdata/ 复制代码 2、配置ha...
hadoop日志存放位置
Alt_way的博客
11-12 1674
日志路径
Hadoop配置项整理
weixin_30500663的博客
07-10 397
记录一下Hadoop的配置和说明，用到新的配置项会补充进来，不定期更新。以配置文件名划分 以hadoop 1.x配置为例 core-site.xml name value Description fs.default.name hdfs://hadoopmaster:9000 定义HadoopMaster的URI和端口 ...
老版本Hadoop-0.20.2资源免费分享
Linux环境下hadoop-0.20.2资源下载 Hadoop是Apache软件基金会开发的一个开源框架，它允许使用简单的编程模型在成百上千的计算机上分布式处理大数据。Hadoop的设计灵感来源于Google的三篇技术论文，特别是关于Google...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值