Timeline Rest API之Tez

Timeline Rest API与 TEZ

通过appId反向获取 queryId 和 sql等信息

可以通过timeline Rest API获取使用tez引擎的application的一些信息

获取dag_id

• /ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3A{$appId}

curl http://ip:port/ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3Aapplication_1689924389107_2946371

# 利用jq直接获取dag_id
curl http://ip:port/ws/v1/timeline/TEZ_DAG_ID?primaryFilter=applicationId%3Aapplication_1689924389107_2946371||jq -r .entities[0].entity

# 获取queryId、status等
jq -r .entities[0].primaryfilters.callerId[0]
jq -r .entities[0].primaryfilters.status[0]

输出主要包含以下信息，均可获取

• entity 该key的值即为dag_id
• starttime dag开始的时间戳
• events 包含dag初始化、提交、开始和结束的时间戳
• primaryfilters：包含dag的一些基本信息，包含hive的queryId、appId、app状态、提交的用户、app的队列
• otherinfo：其他的基本信息

{
   
    "entities":[
        {
   
            "entitytype":"TEZ_DAG_ID",
            "entity":"dag_1689924389107_2946371_1",
            "starttime":1703046180693,
            "events":[
                {
   
                    "timestamp":1703046188967,
                    "eventtype":"DAG_FINISHED",
                    "eventinfo":{
   

                    }
                },
                {
   
                    "timestamp":1703046180944,
                    "eventtype":"DAG_STARTED",
                    "eventinfo":{
   

                    }
                },
                {
   
                    "timestamp":1703046180943,
                    "eventtype":"DAG_INITIALIZED",
                    "eventinfo":{
   

                    }
                },
                {
   
                    "timestamp":1703046180693,
                    "eventtype":"DAG_SUBMITTED",
                    "eventinfo":{
   

                    }
                }
            ],
            "domain":"Tez_ATS_application_1689924389107_2946371_1",
            "primaryfilters":{
   
                "dagName":[
                    "insert overwrite tabl...xxx,1,4),'00')\r\n)tmp (Stage-1)"
                ],
                "queueName":[
                    "root.xxxxx"
                ],
                "callerId":[
                    "hive_20231220122237_xxxxxxx"
                ],
                "applicationId":[
                    "application_1689924389107_2946371"
                ],
                "user":[
                    "xxx"
                ],
                "status":[
                    "SUCCEEDED"
                ]
            },
            "relatedentities":{
   

            },
            "otherinfo":{
   
                "completionApplicationAttemptId":"appattempt_1689924389107_2946371_000001",
                "numFailedTaskAttempts":0,
                "callerContext":"HIVE",
                "numKilledTaskAttempts":0,
                "numCompletedTasks":2,
                "amWebServiceVersion":"2",
                "inProgressLogsURL_1":"core-xxxx:8042/node/containerlogs/container_e02_1689924389107_2946371_01_000001/xxxxx",
                "numSucceededTasks":2,
                "vertexNameIdMapping":{
   
                    "Map 1":"vertex_1689924389107_2946371_1_00",
                    "Reducer 2":"vertex_1689924389107_2946371_1_01"
                },
                "numFailedTasks":0,
                "timeTaken":8023,
                "diagnostics":"",
                "queueName":"root.xxxxx",
                "applicationAttemptId":"appattempt_1689924389107_2946371_000001",
                "callerType":"HIVE_QUERY_ID",
                "startTime":1703046180944,
                "callerId":"hive_20231220122237_xxxxxxx",
                "endTime":1703046188967,
                "applicationId":"application_1689924389107_2946371",
                "numKilledTasks":0,
                "initTime":1703046180943,
                "user":"xxxxx",
                "status":"SUCCEEDED"
            }
        }
    ]
}

获取sql

• /ws/v1/timeline/TEZ_DAG_EXTRA_INFO/{$dag_id}

curl http://ip:port/ws/v1/timeline/TEZ_DAG_EXTRA_INFO/{
   $dag_id}

输出主要包含以下信息

• entity：即dag_id
• starttime dag开始的时间戳
• events：dag提交和开始的时间戳
• otherinfo：一些统计指标及一些其他信息

curl http://ip:port/ws/v1/timeline/TEZ_DAG_EXTRA_INFO/dag_1689924389107_2946371_1

# 利用jq命令直接获取
curl http://ip:port/ws/v1/timeline/TEZ_DAG_EXTRA_INFO/dag_1689924389107_2946371_1|jq -r .otherinfo.dagPlan.dagContext.description

{
   
	"entitytype"