sqoop导hbase zookeeper出现的问题

本文记录了在测试环境中使用SQoop将数据从Oracle导入HBase时遇到的连接拒绝错误。该问题源于RegionServers错误地向Master报告其IP地址为127.0.0.1。通过正确配置/etc/hosts文件解决了此问题。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

由于测试环境的改变,使用sqoop从oracle往hbase导数时一直出现下列错误:

13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/dw/hadoop-2.0.0-cdh4.3.0/lib/native
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-358.el6.x86_64
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:user.name=hadoop
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
13/10/12 10:16:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=180000 watcher=hconnection
13/10/12 10:16:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 7576@master1
13/10/12 10:16:54 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:54 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/10/12 10:16:54 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
13/10/12 10:16:54 INFO util.RetryCounter: Sleeping 2000ms before retry #1...
13/10/12 10:16:55 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:55 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/10/12 10:16:56 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:56 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/10/12 10:16:56 WARN zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
13/10/12 10:16:56 INFO util.RetryCounter: Sleeping 4000ms before retry #2...
13/10/12 10:16:57 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:57 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/10/12 10:16:58 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:58 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
13/10/12 10:16:59 INFO zookeeper.ClientCnxn: Opening socket connection to server 127.0.0.1/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error)
13/10/12 10:16:59 WARN zookeeper.ClientCnxn: Session 0x0 for server null, unexpected error, closing socket connection and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)


导致这个问题的原因是:Master认为Region servers的IP为127.0.0.1 即localhost;Region servers错误的告诉Master它们的IP为127.0.0.1。

解决办法:
在Region servers上修改/etc/hosts
127.0.0.1 fully.qualified.regionservername regionservername localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

修改为:
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6

上述问题解决。
### 使用 Sqoop 操作 HBase 的数据出 #### 数据入方案 当使用 Sqoop 将 MySQL 数据入到 HBase 时,其核心机制是先将数据存储到 HDFS 上,然后再加载到 HBase 表中。因此,在执行此操作之前,需确保 HBase 处于完全分布式模式下运行[^1]。 以下是实现该功能的具体方法: ```bash sqoop import \ --connect jdbc:mysql://<MySQL_HOST>:<PORT>/<DATABASE_NAME> \ --username <USERNAME> \ --password <PASSWORD> \ --table <MYSQL_TABLE_NAME> \ --hbase-table <HBASE_TABLE_NAME> \ --column-family <COLUMN_FAMILY_NAME> \ --hbase-row-key <ROW_KEY_COLUMN> ``` - `--connect`: 提供 JDBC URL 来连接 MySQL 数据库。 - `--username` 和 `--password`: 登录 MySQL 所需的用户名和密码。 - `--table`: 要从 MySQL 入的表名。 - `--hbase-table`: 对应的目标 HBase 表名称。 - `--column-family`: 列族名称。 - `--hbase-row-key`: 设置作为 HBase row key 的字段。 例如,假设有一个名为 `smq_to_hbase` 的 MySQL 表,并希望将其入到 HBase 表 `smq_hbase` 中,则可以按照如下命令执行[^4]: ```bash sqoop import \ --connect jdbc:mysql://192.168.220.20:3306/test \ --username root \ --password 123456 \ --table smq_to_hbase \ --hbase-table smq_hbase \ --column-family info \ --hbase-row-key id ``` #### 数据出方案 由于 Sqoop 不支持直接从 HBase 出数据到 MySQL,通常需要借助 Hive 作为中介工具完成这一过程[^2]。具体流程如下: 1. **创建 Hive 外部表映射 HBase 表**: 配置 Hive 表以访问现有的 HBase 表。 ```sql CREATE EXTERNAL TABLE hive_external_table ( id STRING, column_name STRING ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,family:qualifier") TBLPROPERTIES("hbase.table.name"="your_hbase_table"); ``` 2. **转换为内部表并出至文件系统**: 创建一个新的 Hive 内部表并将外部表的内容插入其中。 ```sql CREATE TABLE hive_internal_table AS SELECT * FROM hive_external_table; INSERT OVERWRITE DIRECTORY '/path/to/output' SELECT * FROM hive_internal_table; ``` 3. **利用 Sqoop 出到 MySQL**: 最终通过 Sqoop 命令读取文件系统的输出目录内容,并写回到目标 MySQL 表中。 ```bash sqoop export \ --connect jdbc:mysql://<HOST>:<PORT>/<DBNAME> \ --username <USER> \ --password <PASSWD> \ --export-dir /path/to/output \ --input-fields-terminated-by ',' \ --table <TARGET_MYSQL_TABLE>; ``` 以上即完成了整个从 HBase 经过 Hive 后再转存回 MySQL 的全过程描述。 ### 注意事项 - 确认所有涉及的服务组件版本兼容性良好 (如 Sqoop 版本匹配对应的 Hadoop/HBase)[^3]。 - 开启 Zookeeper 并配置好集群参数以便管理协调节点间通信状态正常工作[^1].
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值