2021-01-29

hive安装配置完成之后无法执行show databases;可以正常的执行创建库和表的操作

hive命令行执行报错：

0: jdbc:hive2://hadoop102:10000> show databases;
Error: java.io.IOException: java.io.IOException: Cannot create an instance of InputFormat class org.apache.hadoop.mapred.TextInputFormat as specified in mapredWork! (state=,code=0)
 

检查hive日志报错：

Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzopCodec not found

 

检查hadoop的mapred-site.xml文件发现是已经配置了lzo的，但是报错确实找不到类，百度搜索日志报错发现是缺少类对应的jar包

wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz

1. export CFLAGS=-m64

2. ./configure -enable-shared -prefix=/usr/local/hadoop/lzo/
3. make && sudo make install

 

[root@localhost ~]# wget https://github.com/twitter/hadoop-lzo/archive/master.zip

export CFLAGS=-m64

export CXXFLAGS=-m64

export C_INCLUDE_PATH=/usr/local/hadoop/lzo/include

export LIBRARY_PATH=/usr/local/hadoop/lzo/lib

mvn clean package -Dmaven.test.skip=true

cd target/native/Linux-amd64-64

tar -cBf - -C lib . | tar -xBvf - -C ~

cp ~/libgplcompression* $HADOOP_HOME/lib/native/

cp target/hadoop-lzo-0.4.18-SNAPSHOT.jar $HADOOP_HOME/share/hadoop/common/

 

安装完lzo之后将/usr/local/hadoop复制到hadoop集群其他的机器中

 

修改配置文件

在hadoop-env.sh中添加

export LD_LIBRARY_PATH=/usr/local/hadoop/lzo/lib

在core-site.xml中添加

<!--启动压缩-->
    <property>
        <name>io.compression.codecs</name>
        <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.Lz4Codec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
    </property>

    <property>
        <name>io.compression.codec.lzo.class</name>
        <value>com.hadoop.compression.lzo.LzoCodec</value>
    </property>
 

mapred-site.xml中添加

  <property>
    <name>mapreduce.map.output.compress</name>
    <value>true</value>
  </property>

  <property>
    <name>mapreduce.map.output.compress.codec</name>
    <value>org.apache.hadoop.io.compress.LzopCodec</value>
  </property>

  <property>
    <name>mapreduce.output.fileoutputformat.compress</name>
    <value>true</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.codec</name>
    <value>org.apache.hadoop.io.compress.LzopCodec</value>
  </property>
  <property>
    <name>mapreduce.output.fileoutputformat.compress.type</name>
    <value>BLOCK</value>
  </property>
之后重启hadoop集群和hive，在执行上面的查询操作恢复正常如下：

参考链接：https://blog.youkuaiyun.com/codeforces/article/details/84728097