shell脚本实现Hive库表迁移_shell迁移hive数据库表-优快云博客

本文链接：https://blog.youkuaiyun.com/docsz/article/details/84935210

本文介绍了一种批量获取Hive所有库的建表语句的方法，并在新集群中重建这些表。通过Shell脚本自动化处理，首先收集所有数据库及其表的详细信息，然后在目标集群上使用Beeline执行DDL语句。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1、获取hive所有库的建表语句

#获取hive所有库的建表语句

#! /bin/bash
mkdir -p ~/hive/tables/tablesDDL
#获取库名
hive -e "show databases;" > ~/hive/databases.txt

sed -i '1,3d' ~/hive/databases.txt
sed -i '$d' ~/hive/databases.txt
cat ~/hive/databases.txt |sed 's/|//g' | sed -e 's/^[ \t]*//g' -e 's/[ \t]*$//g' > ~/hive/dbs.txt

cat ~/hive/dbs.txt
cat ~/hive/dbs.txt |while read db
do
   echo "use ${db};" >> ~/hive/tables/tablesDDL/${db}_tablesDDL.txt
   sed -i "1 i\create database if not exists ${db};"  ~/hive/tables/tablesDDL/${db}_tablesDDL.txt
   hive -e "use $db; show tables;" > ~/hive/tables/${db}_tables.txt
   #获取表名
   sed -i '1,3d' ~/hive/tables/${db}_tables.txt
   sed -i '$d' ~/hive/tables/${db}_tables.txt
   cat ~/hive/tables/${db}_tables.txt |sed 's/|//g' | sed -e 's/^[ \t]*//g' -e 's/[ \t]*$//g' > ~/hive/tables/${db}_tbs.txt
   #获取建表语句
   cat ~/hive/tables/${db}_tbs.txt | while read table
   do
		hive -e "use $db; show create table $table" > ~/hive/tables/tablesDDL/${table}.txt
		sed -i '/ROW FORMAT SERDE/,$d'  ~/hive/tables/tablesDDL/${table}.txt
		#sed -i '1,3d' ~/hive/tables/tablesDDL/${table}.txt
		sed -i '1d' ~/hive/tables/tablesDDL/${table}.txt
		cat ~/hive/tables/tablesDDL/${table}.txt |sed 's/|//g' | sed -e 's/^[ \t]*//g' -e 's/[ \t]*$//g'  >> ~/hive/tables/tablesDDL/${db}_tablesDDL.txt
		echo ";" >> ~/hive/tables/tablesDDL/${db}_tablesDDL.txt
   done
done

2、在新集群中创建hive表

ls  /data/tablesDDL/*.txt |while read line 
do
  beeline -u jdbc:hive2://${hiveserver2主机名}:10000 -n ${username} -p ${password} -f $line  --force=true
done