Hive Metastore Thrift Server Partition压力测试

最新推荐文章于 2024-11-12 21:33:38 发布

原创最新推荐文章于 2024-11-12 21:33:38 发布 · 5.4k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#Hive #hadoop #性能优化

hadoop 同时被 2 个专栏收录

36 篇文章

订阅专栏

Hive

26 篇文章

订阅专栏

测试环境:

虚拟机10.1.77.84

四核Intel(R) Xeon(R) CPU E5649 @ 2.53GHz，8G内存

起单台metastore server

JVM args : -Xmx7000m -XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC

测试表Schema:

table name: metastore_stress_testing

# col_name data_type comment

a string None

# Partition Information

# col_name data_type comment

pt string None

# Detailed Table Information

Database: default

Owner: hadoop

CreateTime: Wed May 29 11:29:30 CST 2013

LastAccessTime: UNKNOWN

Protect Mode: None

Retention: 0

Location: hdfs://10.1.77.86/user/hive9/warehouse/metastore_stress_testing

Table Type: MANAGED_TABLE

Table Parameters:

transient_lastDdlTime1369798170

# Storage Information

SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

InputFormat: org.apache.hadoop.mapred.TextInputFormat

OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

Compressed: No

Num Buckets: -1

Bucket Columns: []

Sort Columns: []

Storage Desc Params:

serialization.format1

测试方法：

Hive在解析一条query的时候，如果不加table partition predicate(比如直接select * from dpdw_traffic_base)，会在生成物理执行计划，物理计划优化环节PartitionPruner 中调用Hive Metastore Client的getPartition(Table)方法，获取整张表的的List<Partition>，由于每个Partition 可以看成一张大表下的子表，里面包含了各自独立的InputFormatClass, OutputFormatClass, Deserializer, Column Schema , StorageDescriptor，Parameters等等，如果表结构比较复杂，column比较多，Partitions又比较多，会造成返回的对象非常大，而且会对metastore server 造成很大压力。

我们通过模拟HiveMetastoreClient的listPartitions方法来测试性能表现，HiveConf中默认设置metastore client socket timeout超时为20秒(hive.metastore.client.socket.timeout)。为了防止测试过程中read timeout ,我们先将这个值设得很大

生成物理执行计划的call stack:

测试代码：

public class MetastoreTestConcurrent {

 public long timeTasks(int nThread) throws Exception {
 final CountDownLatch startGate = new CountDownLatch(1);
 final CountDownLatch endGate = new CountDownLatch(nThread);
 for (int i = 0; i < nThread; i++) {
   Thread t = new Thread(new Runnable() {
     @Override
     public void run() {
         try {
             startGate.await();
             try {
               HiveConf hConf = new HiveConf();
               hConf.setIntVar(ConfVars.METASTORE_CLIENT_SOCKET_TIMEOUT, 100000);
               HiveMetaStoreClient hmsc = new HiveMetaStoreClient(hConf);
               long start = System.currentTimeMillis();
               List<Partition> parts = hmsc.listPartitions("default", "metastore_stress_testing", (short) -1);
               long end = System.currentTimeMillis();
               System.out.println(Thread.currentThread().getName() + " partition_number:" +  
               parts.size() +" duration:" + (end - start));
             } catch (Exception e) {}
             finally {
             endGate.countDown();
             }
         } catch (InterruptedException e) {}
        }
     });
     t.start();
   }
     long start = System.currentTimeMillis();
     startGate.countDown();
     endGate.await();
     long end = System.currentTimeMillis();
     return end - start;
}

     public static void main(String[] args) throws Exception {
      MetastoreTestConcurrent mtc = new MetastoreTestConcurrent();
      System.out.println("total duration :" + mtc.timeTasks(20));
     }
}

测试结果：

hive metastore server 默认启动的时候会启最小minWorkerThreads = 200，最大maxWorkerThreads = 100000 的ThreadPoolExecutor，并开启TCP keepalive = true

单个Thread:

duration=3500 millis

partition number=1000

duration=16874 millis

partition number=5000

duration=53697 millis

partition number=20000

Partition List RamUsage: 46264576 bytes

duration:75809 millis

partition number:27092

duration :245805 millis

partition number:50000

三个thread

Thread Thread-2 partition_number:26596 duration:96902 millis

Thread Thread-0 partition_number:26596 duration:97135 millis

Thread Thread-1 partition_number:26596 duration:97193 millis

total duration :100235 millis

十个thread

Thread Thread-4 partition_number:27952 duration:255172 millis

Thread Thread-2 partition_number:27952 duration:257144 millis

Thread Thread-0 partition_number:27952 duration:257424 millis

Thread Thread-7 partition_number:27952 duration:258388 millis

Thread Thread-5 partition_number:27952 duration:260496 millis

Thread Thread-1 partition_number:27952 duration:261106 millis

Thread Thread-3 partition_number:27952 duration:261173 millis

Thread Thread-6 partition_number:28386 duration:324847 millis

Thread Thread-8 partition_number:28506 duration:356391 millis

Thread Thread-9 partition_number:28507 duration:356861 millis

total duration: 361216 millis

五个Thread:

Thread-1 partition_number:50000 duration: 555805 millis

Thread-3 partition_number:50000 duration:557810 millis

Thread-2 partition_number:50000 duration:558112 millis

Thread-0 partition_number:50000 duration:558480 millis

Thread-4 partition_number:50000 duration:558897 millis

total duration :560753 millis

结论

Hive在单线程环境下调用listPartitions时间基本成线性增长，在Partition数为20000的时候，返回的结果已经将近46MB了，这还是只有一个实体column和一个virutal column的情况下，由于前面提到partition之间是相互独立的，所以如果partition非常多，会crash metastore server，也会crach本地hive client。

在多线程环境下，调用时间明显增加，因为metastore server没有query cache，每一次调用都会变成对于底层数据库的一次查询。

综合来看，partition比较多的话，一个是在compile环节上会一直block在获取partition schema环节上，从而无法提交job，另一个是即使获得了全表下所有partition信息，会生成job splits file，这个文件也会非常大，会撑满jvm heap，也有可能撑满本地磁盘（会先serialize到本地再提交到hdfs上)

ps:

大前天晚上部署的hive metastore server宕机了
从日志上看应该是用户获取partition信息(get_partitions_with_auth_result方法)的时候size太大导致的OOM，看来还是得限制下partition个数

Exception in thread "pool-1-thread-196" java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2786)

        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)

        at org.apache.thrift.transport.TSaslTransport.write(TSaslTransport.java:445)

        at org.apache.thrift.transport.TSaslServerTransport.write(TSaslServerTransport.java:40)

        at org.apache.thrift.protocol.TBinaryProtocol.writeI16(TBinaryProtocol.java:154)

        at org.apache.thrift.protocol.TBinaryProtocol.writeFieldBegin(TBinaryProtocol.java:109)

        at org.apache.hadoop.hive.metastore.api.FieldSchema.write(FieldSchema.java:421)

        at org.apache.hadoop.hive.metastore.api.StorageDescriptor.write(StorageDescriptor.java:1086)

        at org.apache.hadoop.hive.metastore.api.Partition.write(Partition.java:897)

        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partitions_with_auth_result.write(ThriftHiveMetastore.java:33814)

        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34)

        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)

本文链接http://blog.youkuaiyun.com/lalaguozhe/article/details/9070203，转载请注明