Kylin(SQL on HBase)安装及使用介绍
本文档介绍了Apache Kylin的安装过程，包括遇到的UnknownHostException问题及其解决方法。此外，还详细阐述了如何启动Kylin，登录Web UI，创建项目，刷新数据源，构建模型和CUBE，以及进行Build和Monitor操作。
参考
http://kylin.apache.org/docs24/install/index.html
介绍
Apache Kylin™是一个开源的分布式分析引擎，提供Hadoop/Spark之上的SQL查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由eBay Inc. 开发并贡献至开源社区。它能在亚秒内查询巨大的Hive表。
安装
报错Caused by: java.net.UnknownHostException: hdp6:2181: Name or service not known，这个错看了很久，因为hostname和port都是对的，卡了我很久，后来突然发现2019-10-11 20:38:54,715 INFO [Thread-6] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=hdp6:2181:2181,hdp7:2181:2181,hdp8:2181:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@4da855dd这句中所有的zookeeper都多了一个端口，这才意识到问题
Caused by: org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:197)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:87)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:477)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:204)
... 12 more
2019-10-11 20:38:54,715 INFO [Thread-6] zookeeper.ZooKeeper:438 : Initiating client connection, connectString=hdp6:2181:2181,hdp7:2181:2181,hdp8:2181:2181 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@4da855dd
2019-10-11 20:38:54,717 INFO [close-hbase-conn] hbase.HBaseConnection:136 : Closing HBase connections...
2019-10-11 20:38:54,718 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1710 : Closing zookeeper sessionid=0x16db9f171ad0013
2019-10-11 20:38:54,719 ERROR [Thread-6] curator.CuratorZookeeperClient:206 :
java.io.IOException: java.net.UnknownHostException: hdp6:2181: Name or service not known
at org.apache.curator.ConnectionState.close(ConnectionState.java:117)
at org.apache.curator.CuratorZookeeperClient.close(CuratorZookeeperClient.java:202)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.close(CuratorFrameworkImpl.java:309)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock$Factory$1.run(ZookeeperDistributedLock.java:66)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.UnknownHostException: hdp6:2181: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at org.apache.phoenix.shaded.org.apache.zookeeper.client.StaticHostProvider.<init>(StaticHostProvider.java:61)
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:445)
at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:146)
at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94)
at org.apache.curator.HandleHolder.internalClose(HandleHolder.java:128)
at org.apache.curator.HandleHolder.closeAndClear(HandleHolder.java:71)
at org.apache.curator.ConnectionState.close(ConnectionState.java:113)
... 4 more
2019-10-11 20:38:54,742 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
2019-10-11 20:38:54,742 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : Session: 0x16db9f171ad0013 closed
ERROR: Unknown error. Please check full log.翻看源码如下，很明显这里会自动拼装port
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.kylin.common.util;
import java.util.Arrays;
import javax.annotation.Nullable;
import org.apache.commons.lang3.StringUtils;
import org.apache.hadoop.conf.Configuration;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import com.google.common.base.Function;
import com.google.common.collect.Iterables;
/**
* Use reflection to get zookeeper connect string from HBase configuration.
*/
public class ZooKeeperUtil {
private static final Logger logger = LoggerFactory.getLogger(ZooKeeperUtil.class);
private ZooKeeperUtil() {
throw new IllegalStateException("Class ZooKeeperUtil is an utility class !");
}
public static String getZKConnectStringFromHBase() {
Configuration hconf = null;
try {
Class<? extends Object> hbaseConnClz = ClassUtil.forName("org.apache.kylin.storage.hbase.HBaseConnection", Object.class);
hconf = (Configuration) hbaseConnClz.getMethod("getCurrentHBaseConfiguration").invoke(null);
} catch (Throwable ex) {
logger.warn("Failed to get zookeeper connect string from HBase configuration", ex);
return null;
}
final String serverList = hconf.get("hbase.zookeeper.quorum");
final String port = hconf.get("hbase.zookeeper.property.clientPort");
return StringUtils.join(Iterables.transform(Arrays.asList(serverList.split(",")), new Function<String, String>() {
@Nullable
@Override
public String apply(String input) {
return input + ":" + port;
}
}), ",");
}
}
然后翻看HBase的配置文件，之前把ip和port配置在了一起就导致重复拼接
<property>
<name>hbase.zookeeper.quorum</name>
<value>hdp6:2181,hdp7:2181,hdp8:2181</value>
</property>
所以，翻看源码得知需要拆成两个配置，如下：
<property>
<name>hbase.zookeeper.quorum</name>
<value>hdp6,hdp7,hdp8</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
然后再次启动kylin
bin/kylin.sh start发现报错，如下：
2019-10-11 22:25:31,451 INFO [main-SendThread(hdp7:2181)] zookeeper.ClientCnxn:852 : Socket connection established to hdp7/192.168.1.107:2181, initiating session
2019-10-11 22:25:31,452 DEBUG [main] util.ZookeeperDistributedLock:147 : 37487@hdp4 trying to lock /kylin/kylin_metadata/create_htable/kylin_metadata/lock
2019-10-11 22:25:31,500 INFO [main-SendThread(hdp7:2181)] zookeeper.ClientCnxn:1235 : Session establishment complete on server hdp7/192.168.1.107:2181, sessionid = 0x26db9f17207000a, negotiated timeout = 40000
2019-10-11 22:25:31,516 INFO [main-EventThread] state.ConnectionStateManager:228 : State change: CONNECTED
Exception in thread "main" java.lang.IllegalArgumentException: Failed to find metadata store by url: kylin_metadata@hbase
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:99)
at org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:111)
at org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
at org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:92)
... 3 more
Caused by: java.lang.IllegalStateException: Error while peeking at /kylin/kylin_metadata/create_htable/kylin_metadata/lock
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:209)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.isLockedByMe(ZookeeperDistributedLock.java:220)
at org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:352)
at org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:114)
at org.apache.kylin.storage.hbase.HBaseResourceStore.<init>(HBaseResourceStore.java:88)
... 8 more
Caused by: org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /kylin/kylin_metadata/create_htable/kylin_metadata/lock
at org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:302)
at org.apache.curator.framework.imps.GetDataBuilderImpl$4.call(GetDataBuilderImpl.java:291)
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
at org.apache.curator.framework.imps.GetDataBuilderImpl.pathInForeground(GetDataBuilderImpl.java:288)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:279)
at org.apache.curator.framework.imps.GetDataBuilderImpl.forPath(GetDataBuilderImpl.java:41)
at org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.peekLock(ZookeeperDistributedLock.java:204)
... 12 more
2019-10-11 22:25:32,556 INFO [close-hbase-conn] hbase.HBaseConnection:136 : Closing HBase connections...
2019-10-11 22:25:32,557 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1710 : Closing zookeeper sessionid=0x26db9f172070009
2019-10-11 22:25:32,586 INFO [Thread-6] zookeeper.ZooKeeper:684 : Session: 0x26db9f17207000a closed
2019-10-11 22:25:32,587 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
2019-10-11 22:25:32,604 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : Session: 0x26db9f172070009 closed
2019-10-11 22:25:32,604 INFO [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
ERROR: Unknown error. Please check full log.
启动成功
A new Kylin instance is started by hadoop. To stop it, run 'kylin.sh stop'
Check the log at /home/hadoop/kylin/logs/kylin.log
Web UI is at http://hadoop:7070/kylin
界面
用默认的用户名和密码 ADMIN/KYLIN 登陆 Kylin 网站
创建项目
刷新数据源
构建模型
1.Model Info
2.Data Model
3.Dimensions
4.Measures
5.Settings
构建CUBE
1.Cube Info
2.Dimensions
3.Measures
4.Refresh Setting
5.Advanced Setting
6.Configuration Overwrites
7.Overview
Build
Monitor
扫一扫
2513
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
