kettle8.3,linux安装流程,并执行带参数的转换任务示例
1.配置环境变量:
将安装包上传至linux系统中/opt目录下,解压至/opt/kettle8.3目录中
没有安装包的可以去这篇文章下获取:https://editor.youkuaiyun.com/md/?articleId=140797696
# KETTLE
export KETTLE_HOME=/opt/kettle8.3/data-integration
export PATH=${KETTLE_HOME}:$PATH
2.修改配置文件
1.修改log4j.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="false">
<appender name="PENTAHOFILE" class="org.apache.log4j.DailyRollingFileAppender">
<param name="File" value="logs/pdi.log"/>
<param name="Append" value="false"/>
<param name="DatePattern" value="'.'yyyy-MM-dd"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d %-5p [%c] %m%n"/>
</layout>
</appender>
<appender name="pdi-execution-appender" class="org.apache.log4j.RollingFileAppender">
<param name="File" value="logs/pdi.log"/>
<param name="MaxFileSize" value="10MB"/>
<param name="MaxBackupIndex" value="10"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p <%t> %m%n"/>
</layout>
</appender>
<appender name="PENTAHOCONSOLE" class="org.apache.log4j.ConsoleAppender">
<param name="Target" value="System.out"/>
<param name="Threshold" value="INFO"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}] %m%n"/>
</layout>
</appender>
<category name="org.apache.hadoop.io.retry">
<priority value="INFO" />
</category>
<category name="org.pentaho.platform.osgi">
<priority value="INFO" />
</category>
<category name="org.pentaho.platform.engine.core.system.status">
<priority value="INFO"/>
</category>
<logger name="org.pentaho.di.trans.Trans" additivity="false">
<level value="INFO"/>
<appender-ref ref="pdi-execution-appender"/>
</logger>
<logger name="org.pentaho.di.job.Job" additivity="false">
<level value="INFO"/>
<appender-ref ref="pdi-execution-appender"/>
</logger>
<root>
<priority value="ERROR" />
<appender-ref ref="PENTAHOCONSOLE"/>
</root>
</log4j:configuration>
2.修改spoon.sh
找到PENTAHO_DI_JAVA_OPTIONS,添加 -Dfile.encoding=UTF-8
PENTAHO_DI_JAVA_OPTIONS="-Xms4096m -Xmx4096m -Dfile.encoding=UTF-8"
3.安装webkitgtk包
rpm -ivh webkitgtk-2.4.9-1.el7.x86_64.rpm
4.测试是否安装成功
运行./spoon.sh
如图所示,运行成功,成功显示kettle图形界面
5.运行测试示例
任务制作流程,可查看这篇文章:https://blog.youkuaiyun.com/Qiniin/article/details/140797696?
根据文章的流程制作完成后,将任务拷贝到linux系统中,我这里任务存放路径为/opt/kettle_etljob/
执行任务,并查看运行结果
/opt/kettle8.3/data-integration/pan.sh -file:/opt/kettle_etljob/test03_mysql.ktr -param:datadate=20240730 -param:outfile=/opt/kettle_out
参数解释:
-file:转换存放路径
-param:参数名
执行成功:
查看输出文件:
这里可以看到1min前生成了新的文件,到这里则linux中的kettle安装完成!!!
提醒!!!
./spoon.sh需要用root用户打开,使用其他用户执行会报错,博主目前也没有找到解决办法,如果各位有解决办法可以在评论区分享解决办法
报错内容:
org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
at org.eclipse.swt.SWT.error(SWT.java:4621)
at org.eclipse.swt.widgets.Display.createDisplay(Display.java:1038)
at org.eclipse.swt.widgets.Display.create(Display.java:1025)
at org.eclipse.swt.graphics.Device.<init>(Device.java:179)
at org.eclipse.swt.widgets.Display.<init>(Display.java:590)
at org.eclipse.swt.widgets.Display.<init>(Display.java:581)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:667)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
kettle8.3,linux安装流程,并执行带参数的转换任务示例