阿里DataX编译与案例

最新推荐文章于 2025-06-16 09:48:35 发布

原创最新推荐文章于 2025-06-16 09:48:35 发布 · 465 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#python #操作系统 #数据库

阿里开源-DataX 专栏收录该内容

2 篇文章

订阅专栏

下载源码

svn checkout http://code.taobao.org/svn/datax/trunk

-环境

root@datanode158:~# java -version
java version "1.7.0_45"
root@datanode158:~# python -V
Python 2.7.3
root@datanode158:~# ant -version
Apache Ant(TM) version 1.8.2 compiled on December 3 2011
root@datanode158:~# g++ --version
g++ (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
root@datanode158:~# rpm --version
RPM version 4.9.1.1
root@datanode158:~# dos2unix -V
dos2unix 5.3.1 (2011-08-09)
With native language support.
LOCALEDIR: /usr/share/locale

步骤：

1、进入datax的rpm目录：/datax/rpm

在root下运行：rpmbuild --ba t_dp_datax_engine.spec

会出现一堆FileNotFound的问题

RPM build errors:  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/bin  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/conf  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/engine  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/common  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/libs  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/logs  
    File not found: /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64/home/taobao/datax/jobs

开源也不好好修改下源码，搞到一头雾水=，=

修改t_dp_datax_engine.spec如下：

summary: engine provides core scheduler and data swap storage for DataX   
Name: t_dp_datax_engine  
Version: 1.0.0  
Release: 1  
Group: System  
License: GPL  
AutoReqProv: no   
BuildArch: noarch  
  
%define dataxpath  /home/taobao/datax //改成%{buildroot}/home/taobao/datax  
%define vdataxpath /home/taobao/datax //添加，其中vdataxpath下面要用  
  
%description  
DataX Engine provides core scheduler and data swap storage for DataX   
  
%prep  
cd ${OLDPWD}/../  
export LANG=zh_CN.UTF-8  
ant dist  
  
%build  
  
%install  
dos2unix ${OLDPWD}/../release/datax.py  
  
mkdir -p %{dataxpath}/bin  
mkdir -p %{dataxpath}/conf  
mkdir -p %{dataxpath}/engine  
mkdir -p %{dataxpath}/common  
mkdir -p %{dataxpath}/libs  
mkdir -p %{dataxpath}/jobs  
mkdir -p %{dataxpath}/logs  
  
cp ${OLDPWD}/../jobs/sample/*.xml %{dataxpath}/jobs  
cp ${OLDPWD}/../release/*.py %{dataxpath}/bin/  
  
cp -r ${OLDPWD}/../conf/*.properties %{dataxpath}/conf  
cp -r ${OLDPWD}/../conf/*.xml %{dataxpath}/conf  
  
cp -r ${OLDPWD}/../build/engine/*.jar %{dataxpath}/engine  
  
cp -r ${OLDPWD}/../build/common/*.jar %{dataxpath}/common  
cp ${OLDPWD}/../c++/build/libcommon.so %{dataxpath}/common  
  
cp -r ${OLDPWD}/../libs/commons-io-2.0.1.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/commons-lang-2.4.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/dom4j-2.0.0-ALPHA-2.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/jaxen-1.1-beta-6.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/junit-4.4.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/log4j-1.2.16.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/slf4j-api-1.4.3.jar %{dataxpath}/libs  
cp -r ${OLDPWD}/../libs/slf4j-log4j12-1.4.3.jar %{dataxpath}/libs  
  
%post  
chmod -R 0777 %{dataxpath}/jobs //改成chmod -R 0777 %{vdataxpath}/jobs  
chmod -R 0777 %{dataxpath}/logs //改成chmod -R 0777 %{vdataxpath}/logs  
  
%files  
%defattr(0755,root,root)  
%{dataxpath}/bin // 改成%{vdataxpath}/bin  
%{dataxpath}/conf //改成%{vdataxpath}/conf  
%{dataxpath}/engine //改成%{vdataxpath}/engine   
%{dataxpath}/common //改成%{vdataxpath}/common   
%{dataxpath}/libs //改成%{vdataxpath}/libs   
%attr(0777,root,root) %dir %{dataxpath}/logs //改成%attr(0777,root,root)  
 %{vdataxpath}/logs  
%attr(0777,root,root) %dir %{dataxpath}/jobs //改成 %attr(0777,root,root)   %{vdataxpath}/jobs  
  
  
%changelog  
* Fri Aug 20 2010 meining   
- Version 1.0.0

再次编译

Processing files: t_dp_datax_engine-1.0.0-1.noarch  
Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64  
Wrote: /root/rpmbuild/SRPMS/t_dp_datax_engine-1.0.0-1.src.rpm  
Wrote: /root/rpmbuild/RPMS/noarch/t_dp_datax_engine-1.0.0-1.noarch.rpm  
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.y3UwSl  
+ umask 022  
+ cd /root/rpmbuild/BUILD  
+ /bin/rm -rf /root/rpmbuild/BUILDROOT/t_dp_datax_engine-1.0.0-1.x86_64  
+ exit 0

进入：/root/rpmbuild/RPMS/noarch

发布

rpm -ivh t_dp_datax_engine-1.0.0-1.noarch.rpm

至此安装完成！

测试

root@datanode158:~/rpmbuild/RPMS/noarch# python /home/taobao/datax/bin/datax.py -e true
Taobao DataX V1.0 
Data Source List :
0       mysql
1       sqlserver
2       http
3       fake
4       stream
5       oracle
6       hdfs
7       hbase
Please choose [0-7]: 2
Data Destination List :
0       stream
1       mysql
2       hdfs
3       oracle
4       hbase
Please choose [0-4]: 0
Generate /home/taobao/datax/jobs/httpreader_to_streamwriter_1396012010274.xml successfully .

配置/home/taobao/datax/jobs/httpreader_to_streamwriter_1396012010274.xml

<?xml version="1.0" encoding="UTF-8"?>

<jobs>
  <job id="httpreader_to_streamwriter_job">
    <reader>
      <plugin>httpreader</plugin>
      <!--
default:;
description:how to split url
mandatory:false
name:URLDelimiter
-->
      <param key="urldelimiter" value=";"/>
      <!--
default:\t
description:separator to split urls
mandatory:false
name:fieldSplit
-->
      <param key="field_split" value="\t"/>
      <!--
default:UTF-8
range:UTF-8|GBK|GB2312
description:encode
mandatory:false
name:encoding
-->
      <param key="encoding" value="UTF-8"/>
      <!--
default:\N
description:replace this nullString to null
mandatory:false
name:nullString
-->
      <param key="null_string" value="\N"/>
      <!--
range:legal http url
description:url to fetch data
mandatory:true
name:httpURLs
-->
      <param key="httpurls" value="http://www.baidu.com"/>
      <!--
default:1
range:1-100
description:concurrency of the job
mandatory:false
name:concurrency
-->
      <param key="concurrency" value="1"/>
    </reader>
    <writer>
      <plugin>streamwriter</plugin>
      <!--
default:\t
description:seperator to seperate field
mandatory:false
name:fieldSplit
-->
      <param key="field_split" value="\t"/>
      <!--
default:UTF-8
range:UTF-8|GBK|GB2312
description:stream encode
mandatory:false
name:encoding
-->
      <param key="encoding" value="UTF-8"/>
      <!--
range:
description:print result with prefix
mandatory:false
name:prefix
-->
      <param key="prefix" value="baidu"/>
      <!--
default:true
range:
description:print the result
mandatory:false
name:print
-->
      <param key="print" value="true"/>
      <!--
default:
range:
description:replace null with the nullchar
mandatory:false
name:nullchar
-->
      <param key="nullchar" value="hello"/>
      <!--
default:1
range:1
description:concurrency of the job
mandatory:false
name:concurrency
-->
      <param key="concurrency" value="1"/>
    </writer>
  </job>
</jobs>

自动生成的 xml 文件中，有“?” 标识的 value 值，表示此处用户必须配置，其他地方的默认值用户可以根据自己需要作修改

执行

DataX 的运行命令如下:
/home/taobao/datax/bin/datax.py    job.xml
其中/home/taobao/datax/bin/datax.py 是 DataX 命令行的 python 封装，该执行脚本
是整个 DataX 的程序入口，Job.xml 是该 job 的配置文件。

如上述配置，该程序会将百度首页download下来

................百度页面html、js代码

2014-03-28 21:13:02,204 [main] INFO  schedule.Engine - DataX Reader post work begins .
2014-03-28 21:13:02,204 [main] INFO  schedule.Engine - DataX Reader post work ends .
2014-03-28 21:13:02,204 [main] INFO  schedule.Engine - DataX Writers post work begins .
2014-03-28 21:13:02,205 [main] INFO  schedule.Engine - DataX Writers post work ends .
2014-03-28 21:13:02,205 [main] INFO  schedule.Engine - DataX job succeed .
2014-03-28 21:13:02,210 [main] INFO  schedule.Engine - 
DataX starts work at      : 2014-03-28 21:13:00
DataX ends work at        : 2014-03-28 21:13:02
Total time costs          :                  2s
Average byte speed        :              26KB/s
Average line speed        :                1L/s
Total transferred records :                   1
Total discarded records   :                   0