hive transform python

自己写的一个简单例子，用来做话题描述去重，表中的desc字段 “a-b-a-b-b-c”需要去重  
python代码如下：  
#!/usr/bin/python  
import sys  
reload(sys)  
sys.setdefaultencoding('utf8')  
def quchong(desc):  
    a=desc.split('-')  
    return '-'.join(set(a))  
while True:  
        line = sys.stdin.readline()  
        if line == "":  
                break  
        line = line.rstrip('\n')  
        # your process code here  
        parts = line.split('\t')  
        parts[2]=quchong(parts[2])  
        print "\t".join(parts)  
  
下面是转载过来的，比较详细  
二、hive map中字段自增的写法（转）  
  
1、建立表结构  
  
hive> CREATE TABLE t3 (foo STRING, bar MAP<STRING,INT>)  
    > ROW FORMAT DELIMITED  
    > FIELDS TERMINATED BY '/t'  
    > COLLECTION ITEMS TERMINATED BY ','  
    > MAP KEYS TERMINATED BY ':'  
    > STORED AS TEXTFILE;  
OK  
  
   
  
2、建成的效果  
  
hive> describe t3;  
OK  
foo     string  
bar     map<string,int>  
  
   
  
3、生成test.txt  
  
jeffgeng        click:13,uid:15  
  
   
  
4、把test.txt load进来  
  
hive> LOAD DATA LOCAL INPATH 'test.txt' OVERWRITE INTO TABLE t3;  
Copying data from file:/root/src/hadoop/hadoop-0.20.2/contrib/hive-0.5.0-bin/bin/test.txt  
Loading data to table t3  
OK  
  
   
  
load完效果如下  
  
hive> select * from t3;  
OK  
jeffgeng        {"click":13,"uid":15}  
  
   
  
5、可以这样查map的值  
  
hive> select bar['click'] from t3;  
  
...一系列的mapreduce...  
  
OK  
13  
  
   
  
6、编写add_mapper  
  
#!/usr/bin/python  
import sys  
import datetime  
  
for line in sys.stdin:  
    line = line.strip()  
    foo, bar = line.split('/t')  
    d = eval(bar)  
    d['click'] += 1  
    print '/t'.join([foo, str(d)])  
  
   
  
7、在hive中执行  
  
hive> CREATE TABLE t4 (foo STRING, bar MAP<STRING,INT>)  
    > ROW FORMAT DELIMITED  
    > FIELDS TERMINATED BY '/t'  
    > COLLECTION ITEMS TERMINATED BY ','  
    > MAP KEYS TERMINATED BY ':'  
    > STORED AS TEXTFILE;  
  
   
  
hive> add FILE add_mapper.py  
  
   
  
INSERT OVERWRITE TABLE t4  
    > SELECT  
    >   TRANSFORM (foo, bar)  
    >   USING 'python add_mapper.py'  
    >   AS (foo, bar)  
    > FROM t3;  
FAILED: Error in semantic analysis: line 1:23 Cannot insert into target table because column number/types are different t4: Cannot convert column 1 from string to map<string,int>.  
  
   
  
8、为什么会报出以上错误？貌似add_mapper.py的输出是string格式的，hive无法此这种格式的map认出。后查明，AS后边可以为字段强制指定类型  
  
INSERT OVERWRITE TABLE t4  
SELECT  
  TRANSFORM (foo, bar)  
  USING 'python add_mapper.py'  
  AS (foo string, bar map<string,int>)  
FROM t3;  
  
   
  
9、同时python脚本要去除字典转换后遗留下来的空格，引号，左右花排号等  
  
#!/usr/bin/python  
import sys  
import datetime  
  
for line in sys.stdin:  
    line = line.strip('/t')  
    foo, bar = line.split('/t')  
    d = eval(bar)  
    d['click'] += 1  
    d['uid'] += 1  
    strmap = ''  
    for x in str(d):  
        if x in (' ', "'"):  
            continue  
        strmap += x  
    print '/t'.join([foo, strmap])  
  
   
  
10、执行后的结果  
  
hive> select * from t4;  
OK  
jeffgeng        {"click":14,"uid":null}  
Time taken: 0.146 seconds