日常使用linux下面shell脚本使用sqoop同步到hadoop平台数据实例整理

最新推荐文章于 2025-05-27 01:05:16 发布

原创最新推荐文章于 2025-05-27 01:05:16 发布 · 682 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#hadoop #sqoop #hive

本文介绍了一个用于OSPS系统的数据同步Shell脚本，该脚本通过配置文件读取必要的参数，并利用Sqoop工具将指定的数据从本地导出到Oracle数据库中。主要涉及的表包括TI_OSPS_BIE_BATCH_VERFIY和TI_OSPS_BIE_BATCH_SER_REAL，同时详细展示了如何设置数据导出的目录、字段分隔符以及处理空值的方法。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

osps_public_parameters.cfg------------配置文件路径

queuename="osps"
SAP_CLIENT="800"
SQOOP_CMD="/app/sqoop/bin/sqoop"
HIVE_CMD="/app/hive/bin/hive"

etl_osps_public.cfg------------------数据连接信息

username="etl_to_prism"
password="#######"
hostip="osps.db.sfdc.com.cn"
port="1521"
dbname="OSPS"
schema="osps"
connection_url="jdbc:oracle:thin:@${hostip}:${port}/${dbname}"

-------------业务运行shell脚本与sqoop同步逻辑代码#!/bin/sh

SHELL_DIR="/OPT/SHELL/O_ESG_OSPS_CORE/SHELL/PARAM"
LOG_PATH="/LOG/SHELL/O_ESG_OSPS_CORE"

source ${SHELL_DIR}/osps_public_parameters.cfg
source ${SHELL_DIR}/etl_osps_public.cfg
month_code=$((10#$1))
echo month_code = ${month_code}
echo "1---start----------Execute export_fact_all_actual_yx.sh-----TI_OSPS_BIE_BATCH_VERFIY-------------------------"
###export tt_batch_temp and fact_all_actual_yx

${SQOOP_CMD} export -D mapred.job.queue.name=${queuename} --connect ${connection_url} --username ${username} --password ${password} --table ${schema}.TI_OSPS_BIE_BATCH_VERFIY --export-dir /RESULT/O_ESG_OSPS_CORE/tt_batch_temp --columns VERSION,BATCH_TYPE,TOTAL_COUNT,DATA_TYPE --input-fields-terminated-by ',' --input-null-string '\\N' --input-null-non-string '\\N' --bindir /LOG/SHELL/O_ESG_OSPS_CORE --outdir /LOG/SHELL/O_ESG_OSPS_CORE --class-name tt_batch_temp -m 1

echo "2---end----------Execute export_fact_all_actual_yx.sh-----TI_OSPS_BIE_BATCH_VERFIY-------------------------"

if [ $? -ne 0 ] ;then
echo "Execute export_fact_all_actual_yx.sh------- export------TI_OSPS_BIE_BATCH_SER_REAL------failed"
else
echo "3---start----------Execute export_fact_all_actual_yx.sh---------TI_OSPS_BIE_BATCH_SER_REAL----------------"

-------------显示日志连接打印信息同步表中

${SQOOP_CMD} export -D mapred.job.queue.name=${queuename} --connect ${connection_url} --username ${username} --password ${password} --table ${schema}.TI_OSPS_BIE_BATCH_SER_REAL --export-dir /RESULT/O_ESG_OSPS_CORE/fact_all_actual_yx/mont=${month_code} --columns VERSION,PERNR,BEGDA,ENDDA,COST_CENTER,LGART,CREATE_TM,BETRG,FROM_SYSTEM_CODE --input-fields-terminated-by ',' --input-null-string '\\N' --input-null-non-string '\\N' --bindir /LOG/SHELL/O_ESG_OSPS_CORE --outdir /LOG/SHELL/O_ESG_OSPS_CORE --class-name TI_OSPS_BIE_BATCH_SER_REAL -m 1

echo "4---end----------Execute export_fact_all_actual_yx.sh---------TI_OSPS_BIE_BATCH_SER_REAL----------------"

-------------显示日志打印位置

#echo end flag
echo "Execute export_to_osps_ successfuly at $(date '+%Y-%m-%d %H:%M:%S')" >> "${LOG_PATH}/export_to_osps_success.$(date '+%Y%m%d').log"

fi