[Sqoop基础]-- 本地执行sqoop和oozie上执行sqoop命令需要注意

最新推荐文章于 2024-05-31 17:45:56 发布

原创最新推荐文章于 2024-05-31 17:45:56 发布 · 3.8k 阅读

5 ·

CC 4.0 BY-SA版权

文章标签：

#sqoop

Sqoop 专栏收录该内容

4 篇文章

订阅专栏

本文介绍了使用 SQOOP 工具进行数据同步的具体配置方法，包括不同表的数据导出命令示例，如采购订单、在途库存等，并强调了本地命令与 Oozie 上使用的配置差异。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

-------------------------------------------本地sqoop命令，需要修改为'\N'、并且添加sqoop前缀-------------------------------------------
#采购订单
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_JQZ_T_GAS_GET_ORDER_NUM_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_get_order_num_info_s --input-null-string '\N' --input-null-non-string '\N' --fields-terminated-by ''\t'' --update-mode allowinsert --update-key order_num,times

#在途库存
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_JQZ_T_GAS_TOTAL_STORAGE_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_total_storage_info_s --input-null-string '\N' --input-null-non-string '\N' --fields-terminated-by ''\t'' --update-mode allowinsert --update-key station_no,gather_time

#地区信息表
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_JQZ_T_GAS_AREA_DATA_INFO_M --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_area_data_info_m --input-null-string '\N' --input-null-non-string '\N' --fields-terminated-by ''\t'' --update-mode allowinsert --update-key order_id,order_num

#当前销售额和销售单数
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_YKT_T_GAS_ORDER_INFO_M --export-dir /user/hive/warehouse/origin_ennenergy_onecard_lng.db/a_ykt_t_gas_order_info_m --input-fields-terminated-by ''\t'' --update-mode allowinsert --update-key FStationNo,FTradeDateTime

#平均消耗
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_YKT_T_GAS_AVG_USED_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_onecard_lng.db/a_ykt_t_gas_avg_used_info_s --input-fields-terminated-by '\t' --input-null-string '\N' --input-null-non-string '\N' --fields-terminated-by ''\t'' --update-mode allowinsert --update-key FStationNo,calculation_time

#【新增】导出custom对应站点关系
sqoop export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table C_NY_C_CUST_GAS_STATION_M --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/c_ny_c_cust_gas_station_m --input-null-string '\N' --input-null-non-string '\N' --fields-terminated-by ''\t'' --update-mode allowinsert --update-key CUST_STATION_ID,CUSTOMER_ID

-----------------------------------------------oozie上则需要全部使用双引号--------------------------------------------
#采购订单
export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password 123456 --table A_JQZ_T_GAS_GET_ORDER_NUM_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_get_order_num_info_s --input-null-string "\\N" --input-null-non-string "\\N" --fields-terminated-by "\t" --update-mode allowinsert --update-key order_num,times

#在途库存
export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_JQZ_T_GAS_TOTAL_STORAGE_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_total_storage_info_s --input-null-string "\\N" --input-null-non-string "\\N" --fields-terminated-by "\t" --update-mode allowinsert --update-key station_no,gather_time

#地区信息表
export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_JQZ_T_GAS_AREA_DATA_INFO_M --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/a_jqz_t_gas_area_data_info_m --input-null-string "\\N" --input-null-non-string "\\N" --fields-terminated-by "\t" --update-mode allowinsert --update-key order_id,order_num

#当前销售额和销售单数
export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_YKT_T_GAS_ORDER_INFO_M --export-dir /user/hive/warehouse/origin_ennenergy_onecard_lng.db/a_ykt_t_gas_order_info_m --input-fields-terminated-by "\t" --update-mode allowinsert --update-key FStationNo,FTradeDateTime

#平均消耗
export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table A_YKT_T_GAS_AVG_USED_INFO_S --export-dir /user/hive/warehouse/origin_ennenergy_onecard_lng.db/a_ykt_t_gas_avg_used_info_s --input-fields-terminated-by "\t" --input-null-string "\\N" --input-null-non-string "\\N" --fields-terminated-by "\t" --update-mode allowinsert --update-key FStationNo,calculation_time

#【新增】导出custom对应站点关系

export --connect jdbc:mysql://192.168.1.109/testtests?characterEncoding=utf8 --username root --password123456 --table C_NY_C_CUST_GAS_STATION_M --export-dir /user/hive/warehouse/origin_ennenergy_energytrade.db/c_ny_c_cust_gas_station_m --input-null-string "\\N" --input-null-non-string "\\N" --fields-terminated-by "\t" --update-mode allowinsert --update-key CUST_STATION_ID,CUSTOMER_ID