日常使用linux下面shell脚本使用sqoop同步到hadoop平台数据实例整理

本文介绍了一个用于OSPS系统的数据同步Shell脚本,该脚本通过配置文件读取必要的参数,并利用Sqoop工具将指定的数据从本地导出到Oracle数据库中。主要涉及的表包括TI_OSPS_BIE_BATCH_VERFIY和TI_OSPS_BIE_BATCH_SER_REAL,同时详细展示了如何设置数据导出的目录、字段分隔符以及处理空值的方法。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

osps_public_parameters.cfg------------配置文件路径

queuename="osps"
SAP_CLIENT="800"
SQOOP_CMD="/app/sqoop/bin/sqoop"
HIVE_CMD="/app/hive/bin/hive"

etl_osps_public.cfg------------------数据连接信息

username="etl_to_prism"
password="#######"
hostip="osps.db.sfdc.com.cn"
port="1521"
dbname="OSPS"
schema="osps"
connection_url="jdbc:oracle:thin:@${hostip}:${port}/${dbname}"

-------------业务运行shell脚本与sqoop同步逻辑代码#!/bin/sh

SHELL_DIR="/OPT/SHELL/O_ESG_OSPS_CORE/SHELL/PARAM"
LOG_PATH="/LOG/SHELL/O_ESG_OSPS_CORE"

source ${SHELL_DIR}/osps_public_parameters.cfg
source ${SHELL_DIR}/etl_osps_public.cfg
month_code=$((10#$1))
echo month_code = ${month_code}
echo "1---start----------Execute export_fact_all_actual_yx.sh-----TI_OSPS_BIE_BATCH_VERFIY-------------------------"
###export tt_batch_temp and fact_all_actual_yx

${SQOOP_CMD} export -D mapred.job.queue.name=${queuename}  --connect ${connection_url} --username ${username} --password ${password}  --table ${schema}.TI_OSPS_BIE_BATCH_VERFIY --export-dir /RESULT/O_ESG_OSPS_CORE/tt_batch_temp  --columns VERSION,BATCH_TYPE,TOTAL_COUNT,DATA_TYPE --input-fields-terminated-by ',' --input-null-string '\\N'  --input-null-non-string '\\N' --bindir /LOG/SHELL/O_ESG_OSPS_CORE  --outdir /LOG/SHELL/O_ESG_OSPS_CORE --class-name tt_batch_temp -m 1

echo "2---end----------Execute export_fact_all_actual_yx.sh-----TI_OSPS_BIE_BATCH_VERFIY-------------------------"

if [ $? -ne 0 ] ;then
echo "Execute export_fact_all_actual_yx.sh------- export------TI_OSPS_BIE_BATCH_SER_REAL------failed"
else
echo "3---start----------Execute export_fact_all_actual_yx.sh---------TI_OSPS_BIE_BATCH_SER_REAL----------------"

-------------显示日志连接打印信息同步表中

${SQOOP_CMD} export -D mapred.job.queue.name=${queuename}  --connect ${connection_url} --username ${username} --password ${password}  --table ${schema}.TI_OSPS_BIE_BATCH_SER_REAL --export-dir /RESULT/O_ESG_OSPS_CORE/fact_all_actual_yx/mont=${month_code}  --columns VERSION,PERNR,BEGDA,ENDDA,COST_CENTER,LGART,CREATE_TM,BETRG,FROM_SYSTEM_CODE --input-fields-terminated-by ',' --input-null-string '\\N'  --input-null-non-string '\\N' --bindir /LOG/SHELL/O_ESG_OSPS_CORE  --outdir /LOG/SHELL/O_ESG_OSPS_CORE --class-name TI_OSPS_BIE_BATCH_SER_REAL -m 1

echo "4---end----------Execute export_fact_all_actual_yx.sh---------TI_OSPS_BIE_BATCH_SER_REAL----------------"

-------------显示日志打印位置

#echo end flag
echo "Execute export_to_osps_ successfuly at $(date '+%Y-%m-%d %H:%M:%S')" >> "${LOG_PATH}/export_to_osps_success.$(date '+%Y%m%d').log"
 
fi

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值