ETL数据处理

1、HIve端-单高表

ALTER TABLE test_db.demotwo drop PARTITION (date_ymd='20230609');
insert into test_db.demotwo partition(date_ymd = '20230609') 
 select 
    user_id,
    case when 'height' in ('')
     then concat('[',feature_map['height'],']')
     else feature_map['height']
     end height,
  	concat('{', concat_ws(',',concat( '\'sex\'', ':',concat('\'',feature_map['sex'],'\''))),'}') as string_map,
  	concat('{', concat_ws(',',null),'}') as int_map,
  	concat('{', concat_ws(',',null),'}') as float_map,
  	concat('{', concat_ws(',',null),'}') as date_map,
  	concat('{', concat_ws(',',null),'}') as datetime_map,
  	concat('{', concat_ws(',',null),'}') as array_map
from( select 
  user_id
  ,str_to_map(concat_ws('&&&',collect_set(concat_ws(':', lbl_name, setval1))),'&&&') feature_map
from 
(
select 
	entity_id as user_id,
	lbl_name,
	concat_ws(',',collect_set(
		case when lbl_name in ('') then concat('\'',lbl_val,'\'')
		else lbl_val
		end))  as setval1
FROM 
	test_db.high_test ht 
group by entity_id,lbl_name
) table1 group by user_id )table2

2. Hive端 - 多高表

3、Hive端 - 单宽表

4、Hive端 - 多宽表

5、Hive端 - 多表（不区分高宽）