如何扩展Hive Metastore Thrift RPC服务接口

Hive 在经历十几年的发展,已经获得广泛应用,随着版本升级,不同版本之间的协议接口会发生一些变化,尽管HMS在尽量保持向前兼容,但在大版本变更时,仍然不能保证完全兼容,比如HMS2到HMS3,有关索引(Index)相关的接口变成了约束(constraint),比如下图中左边是HMS3接口定义,右边是HMS2接口定义:

737ecfd6a5134df9ca7965413bf08b56.png

这样当使用Hive 2的Client去访问HMS 3 的服务时候,就会报无效方法(Invalid Method)异常,此时可以通过一个Hive Metastore代理服务来解决。

在具体介绍Hive Metastore代理服务前,需要了解下Hive Metastore服务处理流程:

HMSHandler baseHandler = new HiveMetaStore.HMSHandler("metaserver", conf,false);
IHMSHandler handler = newRetryingHMSHandler(baseHandler, conf);
TProcessor processor = new ThriftHiveMetastore.Processor<>(handler));
final TThreadPoolServer.Args serverArgs = new TThreadPoolServer
    .Args(serverTransport)
    .transportFactory(new TTransportFactory())
    .protocolFactory(new TBinaryProtocol.Factory())
    .processor(processor);


TServer server = new TThreadPoolServer(serverArgs);
server.setServerEventHandler(getServerEventHandler());
server.serve();

上述示例代码启动了Hive Metastore的服务,其中两个重要的组件:

  • TThreadPoolServer:负责请求的监听和派遣

  • TProcessor:负责执行各RPC接口的实现逻辑

TThreadPoolServer功能介绍

TThreadPoolServer主要负责请求的监听和派遣,当接收到新的请求时,在内部将每个client的请求放入Worker线程池,然后继续下一个请求的监听,而工作线程负责一对一处理每个client的请求,直至处理完毕:

#TThreadPoolServer.class
public class TThreadPoolServer extends TServer {
    //private 
2025-04-03 00:03:39: Starting Hive Metastore Server org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083. at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:87) at org.apache.hadoop.hive.metastore.utils.SecurityUtils.getServerSocket(SecurityUtils.java:253) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8979) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8854) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:323) at org.apache.hadoop.util.RunJar.main(RunJar.java:236) Caused by: java.net.BindException: Address already in use (Bind failed) at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:513) at java.net.ServerSocket.bind(ServerSocket.java:375) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:106) ... 11 more Exception in thread "main" org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083. at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91) at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:87) at org.apache.hadoop.hive.metastore.utils.SecurityUtils.getServerSocket(SecurityUtils.java:253) at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8979) at org.apache.hadoop.hive.metastor
最新发布
04-04
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值