Hadoop之TaskTraker分析

最新推荐文章于 2023-09-12 11:20:24 发布

ToBeAndNotToBe

最新推荐文章于 2023-09-12 11:20:24 发布

阅读量3.9k

点赞数

CC 4.0 BY-SA版权

分类专栏： hadoop java 文章标签： hadoop exception string localization jvm events

本文链接：https://blog.youkuaiyun.com/ToBeAndNotToBe/article/details/7188819

TaskTracker在Hadoop中的角色主要包括维护、申请和监控Task，通过与JobTracker的通信来执行任务。初始化过程涉及配置读取、任务与作业管理、JVM管理以及启动健康监测和任务发射等步骤。它会清除原有用户本地文件，新建目录，并监听任务完成事件。TaskController启动后，TaskLauncher不断循环获取并启动新任务。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

TaskTracker的工作职责之前已经和大家提过，主要负责维护，申请和监控Task，通过heartbeat和JobTracker进行通信。

TaskTracker的init过程：

1.读取配置文件，解析参数

2.将TaskTraker上原有的用户local files删除并新建新的dir和file

3. Map<TaskAttemptID, TaskInProgress> tasks = new HashMap<TaskAttemptID, TaskInProgress>(); 清除map

4. this.runningTasks = new LinkedHashMap<TaskAttemptID, TaskInProgress>();记录task的链表
this.runningJobs = new TreeMap<JobID, RunningJob>();记录job的id信息

5.初始化JVMManager：

  mapJvmManager = new JvmManagerForType(tracker.getMaxCurrentMapTasks(), 
        true, tracker);
    reduceJvmManager = new JvmManagerForType(tracker.getMaxCurrentReduceTasks(),
        false, tracker);

6.初始化RPC，获取JobTracker client用于heartbeat通信；

7.new一个后台线程用于监听map完成的事件

  
    this.mapEventsFetcher = new MapEventsFetcherThread();
    mapEventsFetcher.setDaemon(true);
    mapEventsFetcher.setName(
                             "Map-events fetcher for all reduce tasks " + "on " + 
                             taskTrackerName);
    mapEventsFetcher.start();

后台线程的run方法如下：

 while (running) {
        try {
          List <FetchStatus> fList = null;
          synchronized (runningJobs) {
            while (((fList = reducesInShuffle()).size()) == 0) {
              try {
                runningJobs.wait();
              } catch (InterruptedException e) {
                LOG.info("Shutting down: " + this.getName());
                return;
              }
            }
          }
          // now fetch all the map task events for all the reduce tasks
          // possibly belonging to different jobs
          boolean fetchAgain = false; //flag signifying whether we want to fetch
                                      //immediately again.
          for (FetchStatus f : fList) {
            long currentTime = System.currentTimeMillis();
            try {
              //the method below will return true when we have not 
              //fetched all available events yet
              if (f.fetchMapCompletionEvents(currentTime)) {
                fetchAgain = true;
              }
            } catch (E