YARN日志配置指南-优快云博客

yarn配置---日志篇

大数据计算任务一般采用spark或者mr提交到yarn 上运行apllication，因为是并发执行的task，所以任务出错的情况下查找log很不方便，此篇文件简单介绍一下yarn上相关的log配置，便于通过查看log分析提交到yarn上的job.jar.

yarn-site.xml中有几个配置项与日志的保存与删除，本地job执行文件的保留与删除有关。若不做配置，日志和job.jar会在application执行结束后被直接删除掉。

配置项	default	description	推荐值
yarn.log-aggregation-enable	false	Whether to enable log aggregation. Log aggregation collects each container's logs and moves these logs onto a file-system, for e.g. HDFS, after the application completes. Users can configure the "yarn.nodemanager.remote-app-log-dir" and "yarn.nodemanager.remote-app-log-dir-suffix" properties to determine where these logs are moved to. Users can access the logs via the Application Timeline Server. 执行结束后收集各个container本地的日志	true
yarn.log-aggregation.retain-seconds	-1	How long to keep aggregation logs before deleting them. -1 disables. Be careful set this too small and you will spam the name node. 收集的日志的保留时间，以秒为单位，到时后被删除	2592000（30天）
yarn.log.server.url		log server的地址	http://hostname:19888/jobhistory/logs
yarn.nodemanager.local-dirs	${hadoop.tmp.dir}/nm-local-dir	List of directories to store localized files in. An application's localized file directory will be found in: ${yarn.nodemanager.local-dirs}/usercache/${user}/appcache/application_${appid}. Individual containers' work directories, called container_${contid}, will be subdirectories of this. 存放application执行本地文件的根目录，执行完毕后删除，按用户名存储	/data01/yarn/nm-local-dir, /data02/yarn/nm-local-dir, /data03/yarn/nm-local-dir, /data04/yarn/nm-local-dir, /data05/yarn/nm-local-dir, /data06/yarn/nm-local-dir, /data07/yarn/nm-local-dir,、 /data08/yarn/nm-local-dir, /data09/yarn/nm-local-dir, /data10/yarn/nm-local-dir, /data11/yarn/nm-local-dir, /data12/yarn/nm-local-dir
yarn.nodemanager.log-dirs	${yarn.log.dir}/userlogs	Where to store container logs. An application's localized log directory will be found in ${yarn.nodemanager.log-dirs}/application_${appid}. Individual containers' log directories will be below this, in directories named container_{$contid}. Each container directory will contain the files stderr, stdin, and syslog generated by that container. 存放application本地执行日志的根目录，执行完毕后删除，按用户名存储	/data01/yarn/userlogs, /data02/yarn/userlogs, /data03/yarn/userlogs, /data04/yarn/userlogs, /data05/yarn/userlogs, /data06/yarn/userlogs, /data07/yarn/userlogs, /data08/yarn/userlogs, /data09/yarn/userlogs, /data10/yarn/userlogs, /data11/yarn/userlogs, /data12/yarn/userlogs
yarn.nodemanager.log.retain-second	10800	Time in seconds to retain user logs. Only applicable if log aggregation is disabled 日志的保留时间,log aggregation没有enable时，有效	604800
yarn.nodemanager.remote-app-log-dir	/tmp/logs	Where to aggregate logs to. 聚合日志后在hdfs的存放地址	/home/yarn/containerlogs
yarn.nodemanager.remote-app-log-dir-suffix	logs	The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam} 集合日志后的存放地址由 ${remote-app-log-dir}/${user}/{thisParam}构成	logs
yarn.nodemanager.delete.debug-delay-sec	0	Number of seconds after an application finishes before the nodemanager's DeletionService will delete the application's localized file directory and log directory. To diagnose Yarn application problems, set this property's value large enough (for example, to 600 = 10 minutes) to permit examination of these directories. After changing the property's value, you must restart the nodemanager in order for it to have an effect. The roots of Yarn applications' work directories is configurable with the yarn.nodemanager.local-dirs property (see below), and the roots of the Yarn applications' log directories is configurable with the yarn.nodemanager.log-dirs property (see also below). application执行结束后延迟10min删除本地文件及日志	86400