kylin中的metadata详解

kylin的metadata是kylin的核心，也是学习kylin的关键一步。

TASK1

array["0"]="0"

for all in `ls`;

do cubetime=`grep create_time_utc $all | tail -1 | awk '{print $3}' | awk -F, '{print $1}'`;

cubetime=`expr $cubetime / 1000`; 

month=`date -d @$cubetime +%Y%m`;

array["$month"]="0"

done

 

for all in `ls`;

do cubetime=`grep create_time_utc $all | tail -1 | awk '{print $3}' | awk -F, '{print $1}'`;

cubetime=`expr $cubetime / 1000`; 

month=`date -d @$cubetime +%Y%m`;

for i in "${!array[@]}"

do 

if [ $month == $i ];then

newtemp=`expr ${array["$i"]} + 1`

array["$month"]="$newtemp";

break;

fi;

done

done

 

for i in "${!array[@]}"; do printf "%s\t创建cube数：%s\n" "$i" "${array[$i]}"; done

 

Task2

month_ago=`date -d '1 month ago' "+%Y-%m-%d %H:%M:%S"`

timeStamp=`date -d "$month_ago" +%s`

TimeStamp=$((timeStamp*1000+`date "+%N"`/1000000))

for all in `ls`

do  time=`grep create_time_utc $all | tail -1 | awk '{print $3}' | awk -F, '{print $1}'`

if [ $time -gt $TimeStamp ]

then

cubeName=`grep name $all | head -1 | awk '{print $3}' | awk -F, '{print $1}'`

echo $cubeName

fi

done

 

 

Task3

month_ago=`date -d '1 month ago' "+%Y-%m-%d %H:%M:%S"`

timeStamp=`date -d "$month_ago" +%s`

TimeStamp=$((timeStamp*1000+`date "+%N"`/1000000))

for all in `ls`

do

time=`grep last_build_time $all | tail -1 | awk '{print $3}' | awk -F, '{print $1}'`

if [ -n "$time" ]

then

if [ $time -lt $TimeStamp ]

then

cubeName=`grep name $all | head -1 | awk '{print $3}' | awk -F, '{print $1}'`

echo $cubeName

fi

fi

done

 

 

 

Task4&Task5  可以根据status 获取相应的cube 如 disable 和 broken

 

和查询相关的信息都在 system cube 里面

Get top 10 queried cubes

 

 

Get cubes which are not queried in last month, in last three months, respectively

Get cubes which has new segments, but without related jobs for the new segments

 

for all in `ls`;

do

jobid=`grep -A 5 NEW $all | grep last_build_job_id |awk '{print $3}' | awk -F, '{print $1}'`;

if [ -n $jobid ];

then echo $jobid;

fi;

done

 

 

Get cubes which has more than 40 segments

 

count=0

for all in ` grep storage_location_identifier  AB_TEST_CUBE_V11.json`;

do count=`expr $count + 1`;

done

count=`expr $count / 3`;

if [ $count -gt 40 ];

then echo 总 共有$count segment;

fi;

 

 

Task10、