空中的鱼

tiate failed at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297) at org.apache.thrift.transport.TSaslClientT

1456select count(1) from s_ods_trade_full53049SELECT count(1) FROM s_ods_trade a left outer JOIN s_ods_trade_full b ON (a.dp_id = b.dp_id AND a.tid = b.tid and a.part='2012-10-31' and b.part='2012-10-31');104343SELECT count(1) FROM s_ods_trad

配置hive gateway机器Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate fa...

hive同时load数据时，即使是不同分区，也会lock表，[code="java"]conflicting lock present for ymds_logs@ymds_hadoop mode EXCLUSIVE[/code]http://www.cnblogs.com/ggjucheng/archive/2013/01/16/2862749.html解决方法：临时设置...

整理一下，用的时候照着贴就行了。1.hive数据导出 将select的结果放到本地文件系统中INSERT OVERWRITE LOCAL DIRECTORY '/tmp/reg_3' SELECT a.* FROM events a;将select的结果放到hdfs文件系统中INSERT OVERWRITE DIRECTORY '/tmp/hdfs_out' SELEC...

hive计算时找不到文件2014-03-03 11:13:34,585 INFO org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:02014-03-03 11:13:34,585 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished. ...

hive官方https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView说Prior to Hive 0.6.0, lateral view did not support the predicate push-down optimization. In Hive 0.5.0 and earlier,...

接上一篇，对hbase参数进行优化，主要是调整与查询效率相关的参数[b]count[/b][code="sql"]select count(1) from hbase_table;[/code][b]部分字段切表[/b][code="sql"]insert overwrite table hive_table select a,b,c,d from hbase_table;[/co...

[b]前言[/b]我们考虑为什么要将hive与hbase结合，出于hbase有唯一主键的缘故可以将其作为存储，顺手能帮我们完成海量数据“去重”的etl处理，而hive能作为我们统计计算。两者结合简直就是双剑合臂～～，但效果怎么样呢？[b]目的[/b]基于多种hiveSQL去重方式失败后，考虑过使用mysql做近期数据去重的方式，然而不能满足全量去重的业务需求，所以考虑使用hbase的唯...

图1原因： 根据测试，split分割数据以后获取n+m个数值，与预期的n个数值不对应，split默认以空格分割数据，而hive录入数据默认以制表符“\t”分割。解决： for line in sys.stdin: (tid, dp_id, customerno,...此处省略多个字段，共n个字段) = line.strip().split(“\t”)python print输出数据类型格式不准确源码： print

e.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /tmp/hive-maintain/hive_2012-11-28_22-39-43_810_1689858262130334284/_task_tmp.-ext-10002/part=33436268

[code="xml"] hive.aux.jars.path file:///usr/local/hive-0.9.0/lib/hive-hbase-handler-0.9.0.jar,file:///usr/local/hive-0.9.0/lib/hbase-0.94.2.jar,file:///usr/local/hive-0.9.0/lib/zookeeper-3.3.5.jar[/code]将hbase-0.94.2.jar 、zookeeper-3.3.5.

/01/16/2862749.html解决方法：临时设置set hive.support.concurrency=false;

lect的结果放到hdfs文件系统中INSERT OVERWRITE DIRECTORY '/tmp/hdfs_out' SELECT a.* FROM table_name a WHERE a.ds=''; 2.hive数据导入LOAD DATA [LOCAL] INPATH ‘/data/userdata’ [OVERWRITE] INTO TABLE user;#创建表的时候直接指定路径CREATE EXTERNAL TABLE page_view(viewTime INT

or: 0 finished. closing... 2014-03-03 11:13:34,585 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 94880 rows2014-03-03 11:13:34,585 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing... 2014-03-03 11:13:34,585

rlier, if you used a WHERE clause your query may not have compiled. A workaround was to add set hive.optimize.ppd=false; before your query. The fix was made in Hive 0.6.0; see https://issues.apache.org/jira/browse/HIVE-1056: Predicate push down does not wo

sert overwrite table hive_table select a,b,c,d from hbase_table;[/code][b]全字段切表[/b][code="sql"]insert into table test_table partition(part='aa') select * from hbase_table;[/code][b]hive至hive切表[/b][code="sql"]create table test_ta

:org.apache.hadoop.hive.hbase.HBaseSplit at org.apache.hadoop.hive.ql.io.HiveInputFormat$HiveInputSplit.readFields(HiveInputFormat.java:146) at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization

name --column-family info --split-by tid[/code][b]hbae创建表语句[/b][code="sql"]create 'trade','info'[/code][b]hive创建表语句[/b][code="sql"]CREATE EXTERNAL TABLE hbase_trade(tid string,dp_id string,created string,modified s

hive关联hbase表，使用select count(*) from tablename报如下异常：[code="html"]java.io.IOException: Cannot create an instance of InputSplit class = org.apache.hadoop.hive.hbase.HBaseSplit:org.apache.hadoop.hive....

[b]前言[/b]最近在做数据的去重，想到一种类似于关系型数据库rank函数的方式，即获取topN（N=1）的方式，sql步骤如下：1.数据分组，组内排序2.对分组数据进行标记rank3.获取rank...

hive官方https://cwiki.apache.org/confluence/display/Hive/LanguageManual+LateralView说Prior to Hive 0.6.0, lateral view did not support the predicate push-down optimization. In Hive 0.5.0 and earlier, i

接上一篇，对hbase参数进行优化，主要是调整与查询效率相关的参数countselect count(1) from hbase_table;部分字段切表insert overwrite table hive_table select a,b,c,d from hbase_table;全字段切表insert into table test_table partition(part='a

前言我们考虑为什么要将hive与hbase结合，出于hbase有唯一主键的缘故可以将其作为存储，顺手能帮我们完成海量数据“去重”的etl处理，而hive能作为我们统计计算。两者结合简直就是双剑合臂～～，但效果怎么样呢？目的基于多种hiveSQL去重方式失败后，考虑过使用mysql做近期数据去重的方式，然而不能满足全量去重的业务需求，所以考虑使用hbase的唯一主键来达到全量“去重”的效果。

hive关联hbase表，使用select count(*) from tablename报如下异常：java.io.IOException: Cannot create an instance of InputSplit class = org.apache.hadoop.hive.hbase.HBaseSplit:org.apache.hadoop.hive.hbase.HBaseSpl

前置条件hadoop hive hbase环境搭建完成。搭建版本hadoop-1.0.3hive-0.9.0hbase-0.94.2zookeeper-3.3.5步骤配置hive xml，配置hbase与hive结合包 hive.aux.jars.path file:///usr/local/hive-0.9.0/lib/hive-hbase-handler-0.9.0.

hive做大数据处理时，遇到如下异常：[Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.......org.apache.hadoop.hive.ql.metadata.HiveException: org

sql调用python获取数据字段个数太多源码：    for line in sys.stdin:       (tid, dp_id, customerno,...此处省略多个字段，共n个字段) = line.strip().split()异常：    图1原因：    根据测试，split分割数据以后获取n+m个数值，与预期的n个数值不对应，split默认以空格分割数据

svn check out hive 工程图1将相关java build path，保证能编译通过，没有红色xxbuild.xml--》run as--》Ant build，successful后，将build/目录下所有jar包拷贝至lib/文件夹下。若ant没有安装或者不想通过ant编译，可以在apache上下载已经编译好的hive工程，将所有jar包拷贝至lib下，注意，hadoop

1.hive压缩hive>set mapred.output.compress=true;                                                 hive> set mapred.compress.map.output=true;                       hive> set hive.exec.compress.output

整理一下，用的时候照着贴就行了。1.hive数据导出 将select的结果放到本地文件系统中INSERT OVERWRITE LOCAL DIRECTORY '/tmp/reg_3' SELECT a.* FROM events a;将select的结果放到hdfs文件系统中INSERT OVERWRITE DIRECTORY '/tmp/hdfs_out' SELECT a.* F

hive同时load数据时，即使是不同分区，也会lock表，conflicting lock present for ymds_logs@ymds_hadoop mode EXCLUSIVEhttp://www.cnblogs.com/ggjucheng/archive/2013/01/16/2862749.html解决方法：临时设置set hive.support.concurren

配置hive gateway机器Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate fai

[b]sqoop语句[/b]以tid为主键，info为列族[code="html"]sqoop import --connect jdbc:mysql://ip/taobao_db --username hive --password 123456 --table mysql_table_name --hbase-table hbase_table_name --column-family...

[b]前置条件[/b]hadoop hive hbase环境搭建完成。[b]搭建版本[/b]hadoop-1.0.3hive-0.9.0hbase-0.94.2zookeeper-3.3.5[b]步骤[/b]配置hive xml，配置hbase与hive结合包[code="xml"] hive.aux.jars.path file:///usr/loca...

hive做大数据处理时，遇到如下异常：[code="html"][Fatal Error] Operator FS_2 (id=2): Number of dynamic partitions exceeded hive.exec.max.dynamic.partitions.pernode.......org.apache.hadoop.hive.ql.metadata.Hive...

sql调用python获取数据字段个数太多源码： for line in sys.stdin: (tid, dp_id, customerno,...此处省略多个字段，共n个字段) = line.strip().split()异常： 图1原因： 根据测试，split分割数据以后获取n+m个数值，与预期的n个数值不对应，spl...