hadoop job & task日志

最新推荐文章于 2025-07-05 09:48:16 发布
原创 最新推荐文章于 2025-07-05 09:48:16 发布 · 249 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#hadoop

This log details the execution of a Hadoop job, including setup, task assignments, and completion of map and reduce tasks, showcasing the job's progress through various stages with detailed metrics.
摘要
# hadoop jar first-hadoop-0.0.1-SNAPSHOT.jar hello.Hello
12/04/16 10:41:26 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/04/16 10:41:26 INFO mapred.FileInputFormat: Total input paths to process : 1
12/04/16 10:41:26 INFO mapred.JobClient: Running job: job_201204161037_0001
12/04/16 10:41:27 INFO mapred.JobClient: map 0% reduce 0%
12/04/16 10:41:42 INFO mapred.JobClient: map 100% reduce 0%
12/04/16 10:41:58 INFO mapred.JobClient: map 100% reduce 100%
12/04/16 10:42:03 INFO mapred.JobClient: Job complete: job_201204161037_0001
12/04/16 10:42:03 INFO mapred.JobClient: Counters: 30
12/04/16 10:42:03 INFO mapred.JobClient: Job Counters
12/04/16 10:42:03 INFO mapred.JobClient: Launched reduce tasks=1
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=18628
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/04/16 10:42:03 INFO mapred.JobClient: Launched map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: Data-local map tasks=2
12/04/16 10:42:03 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=13856
12/04/16 10:42:03 INFO mapred.JobClient: File Input Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Read=1996
12/04/16 10:42:03 INFO mapred.JobClient: File Output Format Counters
12/04/16 10:42:03 INFO mapred.JobClient: Bytes Written=506
12/04/16 10:42:03 INFO mapred.JobClient: FileSystemCounters
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_READ=696
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_READ=2166
12/04/16 10:42:03 INFO mapred.JobClient: FILE_BYTES_WRITTEN=64709
12/04/16 10:42:03 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=506
12/04/16 10:42:03 INFO mapred.JobClient: Map-Reduce Framework
12/04/16 10:42:03 INFO mapred.JobClient: Map output materialized bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Map input records=23
12/04/16 10:42:03 INFO mapred.JobClient: Reduce shuffle bytes=702
12/04/16 10:42:03 INFO mapred.JobClient: Spilled Records=92
12/04/16 10:42:03 INFO mapred.JobClient: Map output bytes=598
12/04/16 10:42:03 INFO mapred.JobClient: Total committed heap usage (bytes)=337780736
12/04/16 10:42:03 INFO mapred.JobClient: CPU time spent (ms)=2040
12/04/16 10:42:03 INFO mapred.JobClient: Map input bytes=1330
12/04/16 10:42:03 INFO mapred.JobClient: SPLIT_RAW_BYTES=170
12/04/16 10:42:03 INFO mapred.JobClient: Combine input records=0
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input records=46
12/04/16 10:42:03 INFO mapred.JobClient: Reduce input groups=2
12/04/16 10:42:03 INFO mapred.JobClient: Combine output records=0
12/04/16 10:42:03 INFO mapred.JobClient: Physical memory (bytes) snapshot=324034560
12/04/16 10:42:03 INFO mapred.JobClient: Reduce output records=46
12/04/16 10:42:03 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1122484224
12/04/16 10:42:03 INFO mapred.JobClient: Map output records=46
#


Job日志

/var/log/hadoop/root/hadoop-root-jobtracker-localhost.log

2012-04-16 10:41:26,685 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001: nMaps=2 nReduces=1 max=-1
2012-04-16 10:41:26,688 INFO org.apache.hadoop.mapred.JobTracker: Job job_201204161037_0001 added successfully for user 'root' to queue 'default'
2012-04-16 10:41:26,689 INFO org.apache.hadoop.mapred.AuditLogger: USER=root IP=127.0.0.1 OPERATION=SUBMIT_JOB TARGET=job_201204161037_0001 RESULT=SUCCESS
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201204161037_0001
2012-04-16 10:41:26,690 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201204161037_0001
2012-04-16 10:41:26,834 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /tmp/hadoop-root/mapred/system/job_201204161037_0001/jobToken
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201204161037_0001 = 1330. Number of splits = 2
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000000 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201204161037_0001_m_000001 has split on node:/default-rack/localhost
2012-04-16 10:41:26,855 INFO org.apache.hadoop.mapred.JobInProgress: job_201204161037_0001 LOCALITY_WAIT_FACTOR=1.0
2012-04-16 10:41:26,856 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 initialized successfully with 2 map tasks and 1 reduce tasks.
2012-04-16 10:41:29,648 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201204161037_0001_m_000003_0' to tip task_201204161037_0001_m_000003, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,703 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000003_0' has completed task_201204161037_0001_m_000003 successfully.
2012-04-16 10:41:35,713 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000000_0' to tip task_201204161037_0001_m_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000000
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201204161037_0001_m_000001_0' to tip task_201204161037_0001_m_000001, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:35,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201204161037_0001_m_000001
2012-04-16 10:41:41,733 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000000_0' has completed task_201204161037_0001_m_000000 successfully.
2012-04-16 10:41:41,736 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000001_0' has completed task_201204161037_0001_m_000001 successfully.
2012-04-16 10:41:41,741 INFO org.apache.hadoop.mapred.JobTracker: Adding task (REDUCE) 'attempt_201204161037_0001_r_000000_0' to tip task_201204161037_0001_r_000000, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:41:56,781 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_r_000000_0' has completed task_201204161037_0001_r_000000 successfully.
2012-04-16 10:41:56,790 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201204161037_0001_m_000002_0' to tip task_201204161037_0001_m_000002, for tracker 'tracker_localhost:localhost/127.0.0.1:40169'
2012-04-16 10:42:02,797 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201204161037_0001_m_000002_0' has completed task_201204161037_0001_m_000002 successfully.
2012-04-16 10:42:02,798 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201204161037_0001 has completed successfully.
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE subfolder at file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,853 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_1334544086637_root_Max+temperature to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,857 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/var/log/hadoop/root/history/job_201204161037_0001_conf.xml to file:/var/log/hadoop/root/history/done/version-1/localhost_1334543846620_/2012/04/16/000000
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000000_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000001_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000002_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_m_000003_0'
2012-04-16 10:42:02,860 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201204161037_0001_r_000000_0'


Task日志

/var/log/hadoop/root/hadoop-root-tasktracker-localhost.log


2012-04-16 10:41:29,682 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000003_0 task's state:UNASSIGNED
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,688 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000003_0 which needs 1 slots
2012-04-16 10:41:29,822 INFO org.apache.hadoop.mapred.JobLocalizer: Initializing user root on this TT.
2012-04-16 10:41:29,902 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_1468721501
2012-04-16 10:41:29,903 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_1468721501 spawned.
2012-04-16 10:41:29,908 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000003_0/taskjvm.sh
2012-04-16 10:41:30,643 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_1468721501 given task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:34,068 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000003_0 0.0% setup
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000003_0 is done.
2012-04-16 10:41:34,070 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000003_0 was -1
2012-04-16 10:41:34,071 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:34,126 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds.
2012-04-16 10:41:34,178 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName root for UID 0 from the native implementation
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000001_0 task's state:UNASSIGNED
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,716 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000000_0 which needs 1 slots
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_m_000003_0
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000003_0 not found in cache
2012-04-16 10:41:35,717 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,720 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 1 and trying to launch attempt_201204161037_0001_m_000001_0 which needs 1 slots
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-2010691913
2012-04-16 10:41:35,727 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-2010691913 spawned.
2012-04-16 10:41:35,728 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_843978914
2012-04-16 10:41:35,729 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_843978914 spawned.
2012-04-16 10:41:35,730 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000001_0/taskjvm.sh
2012-04-16 10:41:35,732 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/taskjvm.sh
2012-04-16 10:41:37,059 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-2010691913 given task: attempt_201204161037_0001_m_000000_0
2012-04-16 10:41:37,133 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_843978914 given task: attempt_201204161037_0001_m_000001_0
2012-04-16 10:41:40,815 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000000_0 1.0% hdfs://localhost:9000/user/a.txt:0+665
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000000_0 is done.
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000000_0 was 336
2012-04-16 10:41:40,817 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 1
2012-04-16 10:41:40,899 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-2010691913 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:40,907 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000001_0 1.0% hdfs://localhost:9000/user/a.txt:665+665
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000001_0 is done.
2012-04-16 10:41:40,908 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000001_0 was 366
2012-04-16 10:41:40,909 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:40,985 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_843978914 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_r_000000_0 task's state:UNASSIGNED
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,748 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_r_000000_0 which needs 1 slots
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_r_1468721501
2012-04-16 10:41:41,763 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_r_1468721501 spawned.
2012-04-16 10:41:41,765 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/taskjvm.sh
2012-04-16 10:41:43,008 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_r_1468721501 given task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:49,586 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:49,972 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50600, bytes: 336, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000000_0, duration: 20894000
2012-04-16 10:41:49,979 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.0.1:50060, dest: 127.0.0.1:50601, bytes: 366, op: MAPRED_SHUFFLE, cliID: attempt_201204161037_0001_m_000001_0, duration: 1496000
2012-04-16 10:41:50,639 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,654 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is in commit-pending, task state:COMMIT_PENDING
2012-04-16 10:41:50,800 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 0.0% reduce > copy >
2012-04-16 10:41:53,772 INFO org.apache.hadoop.mapred.TaskTracker: Received commit task action for attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:55,609 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,611 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_r_000000_0 1.0% reduce > reduce
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_r_000000_0 is done.
2012-04-16 10:41:55,612 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_r_000000_0 was -1
2012-04-16 10:41:55,617 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:41:55,656 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_r_1468721501 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201204161037_0001_m_000002_0 task's state:UNASSIGNED
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,791 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201204161037_0001_m_000002_0 which needs 1 slots
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: Received KillTaskAction for task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,792 INFO org.apache.hadoop.mapred.TaskTracker: About to purge task: attempt_201204161037_0001_r_000000_0
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201204161037_0001_m_-919394950
2012-04-16 10:41:56,797 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201204161037_0001_m_-919394950 spawned.
2012-04-16 10:41:56,799 INFO org.apache.hadoop.mapred.TaskController: Writing commands to /tmp/hadoop-root/mapred/local/ttprivate/taskTracker/root/jobcache/job_201204161037_0001/attempt_201204161037_0001_m_000002_0/taskjvm.sh
2012-04-16 10:41:57,534 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_201204161037_0001_m_-919394950 given task: attempt_201204161037_0001_m_000002_0
2012-04-16 10:41:57,992 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0%
2012-04-16 10:42:00,964 INFO org.apache.hadoop.mapred.TaskTracker: attempt_201204161037_0001_m_000002_0 0.0% cleanup
2012-04-16 10:42:00,965 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_201204161037_0001_m_000002_0 is done.
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for attempt_201204161037_0001_m_000002_0 was -1
2012-04-16 10:42:00,966 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2012-04-16 10:42:01,023 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201204161037_0001_m_-919394950 exited with exit code 0. Number of tasks it ran: 1
2012-04-16 10:42:02,862 INFO org.apache.hadoop.mapred.TaskTracker: Received 'KillJobAction' for job: job_201204161037_0001
2012-04-16 10:42:02,864 INFO org.apache.hadoop.mapred.IndexCache: Map ID attempt_201204161037_0001_m_000002_0 not found in cache
2012-04-16 10:42:02,872 INFO org.apache.hadoop.mapred.UserLogCleaner: Adding job_201204161037_0001 for user-log deletion with retainTimeStamp:1334630522867


Job用户日志目录
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_m_000000_0/stdout
/tmp/hadoop-root/mapred/local/userlogs/job_201204161037_0001/attempt_201204161037_0001_r_000000_0/stdout
Yarn 资源调度全面解析
qq_39712666的博客
09-22 475
远在 Hadoop v1 版本时期，Yarn 还没有出现，资源管理是和 Hadoop 耦合在一起的，我们没法在 Hadoop 集群运行除了 MapReduce 以外其他计算框架的任务，在策略和管理上也没那么成熟。Yarn 的出现可是说是为 Hadoop 生态圈大数据技术的百花齐放奠定了一个基础，一直到目前为止 Yarn 仍然是在大数据领域最常用的一个资源调度框架，Spark/Flink/Hive ...
Spark中Application、Job、Stage和Task区别详解
优快云 精品推荐
08-18 695
在Spark中，有一些关键的概念，如Application、Job、Stage和Task，它们描述了Spark作业的不同层次和组成部分。下面我将详细介绍每个概念，并通过示例说明它们之间的区别。
Hadoop: Job vs Application vs Task vs Task attempt
sinat_34763749的博客
07-25 1218
In terms of YARN, the programs that are being run on a cluster are called applications. In terms of MapReduce they are called jobs. So, if you are running MapReduce on YARN, job and application are th...
【Hadoop】JobQueueTaskScheduler
TragicJun的专栏
11-08 1205
JobQueueTaskScheduler是JT默认的任务调度器，其本质上维护一个priority-based FIFO作业队列，基本的机制是优先调度高优先级/提交时间早的作业。但是，具体的调度策略需要综合考虑一系列的问题，比如cluster资源利用率、系统吞吐率、任务locality等等。下面从这些方面来分析一下调度策略。 单个TaskTracker利用率 最初，JT调度器每次h
《Hadoop技术内幕》读书笔记——Task运行过程分析
weixin_30952535的博客
10-19 190
本文是董西成的Hadoop技术内幕一书的读书章节总结。 第八章 Task运行过程分析 所有Task需要周期性地向TaskTracker汇报最新进度和计数器值，而这正是由Reporter组件实现的，其中Reporter汇报的信息中包含两个部分：任务执行进度以及任务计数器值。任务执行进度hadoop采用简单的线性模型计算每个阶段的进度值，对于Map Task而言，作为一个大阶段...
大数据开发面试部分：Hadoop 中 job 和 task 之间的区别是什么（大数据开发面试）
wyqwilliam的博客
07-13 5262
JobTracker 是一个 master 服务，软件启动之后 JobTracker 接收 Job，负责调度 Job的每一个子任务， task 运行于 TaskTracker 上，并监控它们，如果发现有失败的 task 就重新运行它。一般情况应该把 JobTracker 部署在单独的机器上。 TaskTracker 是运行在多个节点上的 slaver 服务。TaskTracker 主动与 Job...
Apache Hadoop之历史服务器&日志聚集配置
ningkangming的博客
07-10 735
介绍了Apache Hadoop的分布式集群环境搭建，并测试了MapReduce分布式计算案例。但集群历史做了哪些任务，任务执行日志等信息还需要配置历史服务器和日志聚集才能更好的查看。
Hadoop【使用篇】05：MapReduce任务日志查看
weixin_40612128的博客
03-01 4833
MapReduce任务日志查看
hadoop开启namenode审计日志
快乐程序员的专栏
05-08 8493
hadoop开启namenode审计日志，需要单独配置log4j相关信息，以下是
安装Hadoop集群&入门&源码编译
最新发布
如无必要勿增实体
07-05 1083
本文介绍了Hadoop完全分布式集群的安装步骤。首先准备三台机器分别部署NameNode、ResourceManager和SecondaryNameNode。安装前需配置SSH免密登录和编写集群分发脚本xsync。随后安装JDK并配置环境变量。接着下载解压Hadoop包并设置相关环境变量。集群部署规划明确了各节点角色，避免资源冲突。最后配置核心文件core-site.xml，指定NameNode地址、数据存储目录等参数。
Hadoop和yarn面试题（详解）
weixin_51194902的博客
05-10 481
Hadoop优化有哪些方面 列出正常工作的hadoop集群中hadoop都分别启动哪些进程以及他们的作用 Hadoop总job和Tasks之间的区别是什么？ Hadoop高可用HA模式 简要描述安装配置一个hadoop集群的步骤 fsimage和edit的区别 yarn的三大调度策略 hadoop的shell命令用的多吗?,说出一些常用的 用mr实现用户pv的top10？ 1. Hadoop优化有哪些方面 0）HDFS 小文件影响 （1）影响 NameNode 的寿命，因为文件元数据存储在 Nam..
通过HadoopAPI获取task日志内容
some_321的专栏
05-16 555
 在集群中查看Task日志的方法，一般有两个： 1，通过Hadoop提供的WebConsole，直接在页面中追踪查看； 2，到集群中运行该task的节点上，查看日志文件。每个tasktracker子进程都会用log4j产生三个日志文件，分别是syslog，stdout，stderr。这些日志文件存放到%HADOOP_LOG_DIR%目录下的userlogs的子目录中。但是通过该方法，需要追踪...
task中隐藏不必要日志
oochen_98的博客
11-04 272
logger
计算机领域中Task和Job的区别
lixinkuan的博客
02-16 3812
There are only two hard things in Computer Science: cache invalidation and naming things (计算科学中只有两件事最难：命名和缓存失效) —— Phil Karlton 本文就是讨论一...
Hadoop中job的执行流程
YYDS_emmm的博客
06-26 1240
Hadoop中job的执行流程 （1）作业提交 第 1 步：client 向整个集群提交job,向 ResourceManager 申请一个jobid。 第 2 步：ResourceManager 给 client 返回该 job 资源的提交路径和作业 id。 第 3 步：client 提交 jar 包、切片信息和配置文件到指定的资源提交路径。 第 4 步：client 提交完资源后，向 ResourceManager 申请运行 MrAppMaster。 （2）作业初始化 第 5 步：当 Resource
HDFS
meng984611383的博客
04-17 251
转载：http://www.aboutyun.com/thread-7088-1-1.html http://www.cnblogs.com/sky-sql/p/6881757.html 1.job的本质是什么？ 在MapReduce中，一个准备提交执行的应用程序称为“作业（job）”2.任务的本质是什么？从一个作业划分出 得、运行于各个计算节点的工作单元称为“任务（task）”3.文件系统...
从hadoop发展角度彻底明白hadoop1.x与hadoop2.x的区别
执念的博客
12-13 8038
问题导读 1.hadoop1.x改造如果是两个jobtraker，你认为解决了什么问题？ 2.hadoop1.x改造如果是两个jobtraker，你认为未解决了什么问题？ 3.你如何看待hadoop2.x的yarn 鉴于本人面试提到过这个问题：hadoop1.x和hadoop2.x的区别，起始存在很多模糊的地方，希望看到这篇文章，能够对hadoop有个基本的认识。 转载自http
6. MapReduce中作业（Job）、任务（task）的一生概览
庐州小白的博客
10-27 3068
本节内容完全来自 深入理解大数据：大数据处理与编程实践 机械工业出版社 没啥好说的，直接上内容 一、作业 首先， 用户程序客户端通过作业客户端接口程序JobClient提交一个用户程序。 然后JobClient向JobTracker提交作业执行请求并获得一个Job ID。 JobClient同时也会将用户程序作业和待处理的数据文件信息准备好并存储在HDFS中。 JobClient正式向JobTracker提交和执行该作业。 JobTracker接受并调度该作业，并进行作业的初始化准备工作， 根据待处理数
输出 Hadoop WordCount 日志结果
06-30
Hadoop WordCount程序执行过程中会生成详细的日志信息，这些日志可以帮助用户了解作业的运行状态、调试问题以及优化性能。当执行以下命令启动WordCount任务时： ```bash hadoop jar wc.jar com.yyds.wordcount.WordcountDriver /user/yyds/input /user/yyds/output ``` 系统会在控制台输出一些基本的运行时信息，例如任务进度、Map和Reduce阶段的状态等。此外，完整的日志通常存储在Hadoop的日志目录中，具体路径取决于集群的配置。 ### 日志获取方式 1. **YARN应用程序日志** Hadoop使用YARN作为资源调度框架时，可以通过以下命令查看特定作业的日志： ```bash yarn logs -applicationId <application_id> ``` 其中 `<application_id>` 是作业运行时由ResourceManager分配的唯一标识符，可以在控制台输出中找到该ID。此命令将输出包括ApplicationMaster和各个Container的日志信息[^1]。 2. **HDFS上的日志文件** 如果启用了日志聚合功能（`yarn.log-aggregation-enable` 设置为 `true`），则日志会被上传到HDFS中的指定路径，通常是 `/tmp/logs` 或者根据配置设定的路径。可以使用如下命令访问这些日志： ```bash hadoop fs -cat hdfs://<namenode>:<port>/tmp/logs/<application_id>/* ``` 3. **Web UI 查看日志** YARN提供了基于Web的界面，默认地址为 `http://<resourcemanager-host>:8088`。通过该界面可以浏览所有已提交的应用程序，并点击进入具体的Job页面查看其日志内容。 4. **本地NodeManager日志** 如果未启用日志聚合，则日志文件默认保存在各个NodeManager节点的本地磁盘上，路径通常为 `$HADOOP_LOG_DIR/userlogs/<application_id>`。需要登录到对应的节点并查看相应目录下的日志文件。 5. **自定义日志输出** 在编写MapReduce程序时，可以通过 `org.apache.hadoop.mapreduce.Mapper` 和 `org.apache.hadoop.mapreduce.Reducer` 中的 `Context` 对象记录自定义日志。例如： ```java context.write(new Text("DEBUG"), new IntWritable(1)); // 输出调试信息 ``` 这些信息将在Mapper或Reducer的输出中体现，并可通过上述方式获取。 6. **日志级别控制** Hadoop支持通过配置文件调整日志级别，如 `log4j.properties` 文件中设置 `hadoop.root.logger=DEBUG,console` 可以将日志级别设为DEBUG并将日志输出到控制台，便于调试。 ### 注意事项 - 确保Hadoop集群的 `yarn.nodemanager.remote-app-log-dir` 配置项指向一个有效的HDFS路径。 - 若需长期保留日志，建议定期归档HDFS中的日志文件。 - 使用 `mapreduce.task.timeout` 和 `mapreduce.map.task.failures tolerated` 等参数可优化任务失败处理策略，从而减少无效日志的产生。 通过以上方法，可以有效地获取和分析Hadoop WordCount作业的执行日志，帮助诊断潜在问题并提升作业性能。
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值