kettle环境初始化

本文详细介绍了Kettle启动时的初始化流程,包括创建.kettle目录、读取kettle.properties文件、初始化日志配置、注册插件类型等内容。

背景

程序启动时会初始化kettle的运行环境,例如,spoon启动时main方法就会调用KettleEnvironment.init()来为spoon的运行环境作初始化。通过KettleEnvironment.init()来初始化环境时,会读取属性文件、注册插件等,以init()方法为切入点,调试了一下源码,大概整理了一下脉络,并简单记录下来:

步骤

  1. 调用KettleClientEnvironment.init()方法初始化客户端环境,具体的子流程以下:
  2. 创建.kettle目录,并在该目录下创建一个默认的kttle.properties文件—createKettleHome()
  3. 读取kettle.properties文件,把内容设置到系统变量中—EnvUtil.environmentInit()
  4. 初始化一些日志配置
  5. 注册插件类型(5种类型)并进行初始化—PluginRegistry.init( true )
  6. 如果运行在一个standalone model(例如spoon,kitche,carte),则会初始化JNDI,具体细节查看JndiUtil.initJNDI()
  7. 把本地插件类型RowDistributionPluginType,StepPluginType,PartitionerPluginType,JobEntryPluginType,LogTablePluginType, RepositoryPluginType,LifecyclePluginTypeKettleLifecyclePluginType,ImportRulePluginType,CartePluginType,CompressionPluginType, AuthenticationProviderPluginType,AuthenticationConsumerPluginType添加到PluginRestry类的**pluginTypes:List**中
  8. 调用PluginRegistry.init()进行初始化插件
  9. 初始化kettle变量,其实是解析kettle-variables.xml并保存里面的配置到List里—KettleVariablesList.init()
  10. 初始化生命周期监听器—initLifecycleListeners()
  11. 初始化日志插件—initLoggingPlugins()

流程图

Created with Raphaël 2.3.0 KettleEnvironment.init()方法 调用KettleClientEnvironment.init()方法初始化客户端环境 JndiUtil.initJNDI():初始化JNDI,当运行在一个standalone model(spoon,kitche,carte) 注册本地插件类型(13种,如StepPluginType等),并进行初始化PluginRegistry.init() KettleVariablesList.init():初始化kettle变量,其实是解析kettle-variables.xml并保存里面的配置到List里 initLifecycleListeners():初始化生命周期监听器;initLoggingPlugins():初始化日志插件 结束
Created with Raphaël 2.3.0 调用KettleClientEnvironment.init()方法初始化客户端环境 createKettleHome:创建.kettle目录,并在该目录下创建一个默认的kttle.properties文件 EnvUtil.environmentInit():读取kettle.properties文件,把内容设置到系统变量中 初始化一些日志配置 注册插件类型(5种类型)并进行初始化:PluginRegistry.init( true ); 结束
java.nio.file.FileSystemNotFoundException at com.sun.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:171) at com.sun.nio.zipfs.ZipFileSystemProvider.getPath(ZipFileSystemProvider.java:157) at java.nio.file.Paths.get(Paths.java:143) at com.ruoyi.kettle.tools.KettleUtil$KettleEnv.init(KettleUtil.java:448) at com.ruoyi.kettle.tools.KettleUtil.runKettleJob(KettleUtil.java:238) at com.ruoyi.kettle.service.impl.KettleJobServiceImpl.runJobRightNow(KettleJobServiceImpl.java:259) at com.ruoyi.kettle.tools.RedisStreamUtil.readGroup(RedisStreamUtil.java:251) at com.ruoyi.kettle.tools.CommandLineRunnerImpl$1.run(CommandLineRunnerImpl.java:26) 16:51:35.413 [Thread-3] ERROR c.r.k.t.KettleUtil - [init,461] - Kettle环境初始化失败 java.nio.file.FileSystemNotFoundException public static class KettleEnv { public static void init() { try { //增加自定义的插件库 int lengths = StepPluginType.getInstance().getPluginFolders().size(); URL resourceUrl = KettleUtil.class.getClassLoader().getResource(""); if (resourceUrl == null) { throw new FileNotFoundException("无法定位资源目录"); } Path homePath = Paths.get(resourceUrl.toURI()); Path kettleRoot = homePath.resolve("kettlePlugins").normalize(); String pluginPath = String.valueOf(kettleRoot); log.info("kettle插件路径,{}",pluginPath); if (!StepPluginType.getInstance().getPluginFolders().stream().anyMatch(a -> a.getFolder().equals(pluginPath))) { StepPluginType.getInstance().getPluginFolders().add(new PluginFolder(pluginPath, false, true)); } KettleEnvironment.init(); EnvUtil.environmentInit(); log.info("Kettle环境初始化成功"); } catch (Exception e) { e.printStackTrace(); log.error("Kettle环境初始化失败"); } } } kettle插件在D:\IDEA\java_kettle\smart-kettle\RuoYi-master\ruoyi-admin\src\main\resources\kettlePlugins,我希望执行kettle作业时候直接用本地程序的插件,ruoyi框架打jar包部署,提示上面的情况,怎么调整。
06-06
package com.ruoyi.kettle.config; import com.ruoyi.kettle.tools.KettleUtil; import lombok.extern.slf4j.Slf4j; import org.pentaho.di.core.Const; import org.pentaho.di.core.KettleEnvironment; import org.pentaho.di.core.exception.KettleException; import org.pentaho.di.core.plugins.PluginFolder; import org.pentaho.di.core.plugins.StepPluginType; import org.pentaho.di.core.util.EnvUtil; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import java.io.File; import java.io.FileNotFoundException; import java.io.IOException; import java.io.InputStream; import java.net.URISyntaxException; import java.net.URL; import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; import java.nio.file.StandardCopyOption; import java.util.Enumeration; import java.util.concurrent.locks.ReentrantReadWriteLock; import java.util.jar.JarEntry; import java.util.jar.JarFile; @Slf4j public class KettleEnv { private static final String PLUGIN_DIR = "kettlePlugins"; private static final String JNDI_DIR = "simple-jndi"; private static volatile Path extractedPluginPath; private static volatile Path extractedJndiPath; private static String pluginBasePath; // 读写锁确保线程安全 private static final ReentrantReadWriteLock jndiLock = new ReentrantReadWriteLock(); public static synchronized void init() { try { String pluginPath = getConfiguredPluginPath(); if (pluginPath == null) { // 自动检测并处理插件目录 pluginPath = resolvePluginPath(); } log.info("使用的Kettle插件路径: {}", pluginPath); // 添加插件路径(避免重复添加) String finalPluginPath = pluginPath; if (StepPluginType.getInstance().getPluginFolders().stream() .noneMatch(a -> a.getFolder().equals(finalPluginPath))) { StepPluginType.getInstance().getPluginFolders().add( new PluginFolder(pluginPath, false, true) ); } // 初始化Kettle环境 KettleEnvironment.init(); EnvUtil.environmentInit(); log.info("Kettle环境初始化成功"); pluginBasePath = pluginPath; } catch (Exception e) { log.error("Kettle环境初始化失败", e); } } /** * 设置JNDI环境(任务运行前调用) */ public static void setupJndiEnvironment() { jndiLock.writeLock().lock(); try { Path jndiPath = getDefaultJndiPath(); // 确保目录存在 if (!Files.exists(jndiPath)) { Files.createDirectories(jndiPath); log.info("创建JNDI目录: {}", jndiPath); } // 设置JNDI系统属性 System.setProperty("org.osjava.sj.root", jndiPath.toString()); System.setProperty("org.osjava.sj.delimiter", "/"); System.setProperty("org.osjava.sj.reload", "true"); Const.JNDI_DIRECTORY = jndiPath.toString(); log.info("JNDI路径已设置: {}", jndiPath); } catch (Exception e) { log.error("JNDI环境设置失败", e); throw new RuntimeException(e); } finally { jndiLock.writeLock().unlock(); } } /** * 获取默认JNDI路径 */ public static Path getDefaultJndiPath() { // 优先使用已提取的JNDI路径 if (extractedJndiPath != null) { return extractedJndiPath; } // 检查是否有配置的外部JNDI路径 String externalJndiPath = System.getProperty("kettle.jndi.path"); if (externalJndiPath != null && Files.exists(Paths.get(externalJndiPath))) { return Paths.get(externalJndiPath); } // 默认使用项目根目录下的config/simple-jndi return Paths.get(System.getProperty("user.dir"), "config/simple-jndi"); } /** * 读取jdbc.properties文件内容为字符串(保留原始格式) */ public static String readJdbcPropertiesAsString() throws IOException { return readJdbcPropertiesAsString(getDefaultJndiPath()); } public static String readJdbcPropertiesAsString(Path jndiPath) throws IOException { jndiLock.readLock().lock(); try { Path jdbcFile = jndiPath.resolve("jdbc.properties"); if (!Files.exists(jdbcFile)) { log.warn("jdbc.properties文件不存在: {}", jdbcFile); return ""; } return new String(Files.readAllBytes(jdbcFile), StandardCharsets.UTF_8); } finally { jndiLock.readLock().unlock(); } } /** * 将字符串内容写入jdbc.properties文件(保留原始格式) */ public static void writeJdbcPropertiesFromString(String content) throws IOException { writeJdbcPropertiesFromString(getDefaultJndiPath(), content); } public static void writeJdbcPropertiesFromString(Path jndiPath, String content) throws IOException { jndiLock.writeLock().lock(); try { Path jdbcFile = jndiPath.resolve("jdbc.properties"); // 确保目录存在 if (!Files.exists(jndiPath)) { Files.createDirectories(jndiPath); } Files.write(jdbcFile, content.getBytes(StandardCharsets.UTF_8)); log.info("jdbc.properties更新成功, 路径: {}", jdbcFile); } finally { jndiLock.writeLock().unlock(); } } public static String getPluginBasePath() { return pluginBasePath; } /** * 获取配置的插件路径(外部目录优先) */ public static String getConfiguredPluginPath() { // 1. 检查系统属性 String sysPropPath = System.getProperty("kettle.plugin.path"); if (sysPropPath != null && Files.exists(Paths.get(sysPropPath))) { return sysPropPath; } // 2. 检查当前工作目录下的插件目录 Path workingDirPath = Paths.get(System.getProperty("user.dir"), PLUGIN_DIR); if (Files.exists(workingDirPath)) { return workingDirPath.toString(); } // 3. 检查JAR文件同级的插件目录 Path jarDirPath = null; try { jarDirPath = Paths.get(KettleUtil.class.getProtectionDomain() .getCodeSource().getLocation().toURI()) .getParent() .resolve(PLUGIN_DIR); } catch (URISyntaxException e) { throw new RuntimeException(e); } if (Files.exists(jarDirPath)) { return jarDirPath.toString(); } return null; } /** * 解析插件路径(支持JAR包和IDE环境) */ private static String resolvePluginPath() throws Exception { // 优先返回已提取的路径(避免重复提取) if (extractedPluginPath != null) { return extractedPluginPath.toString(); } URL resourceUrl = KettleUtil.class.getClassLoader().getResource(PLUGIN_DIR); if (resourceUrl == null) { throw new FileNotFoundException("未找到Kettle插件资源目录"); } // 开发环境直接使用路径 if (!"jar".equals(resourceUrl.getProtocol())) { // 同时提取JNDI目录 extractJndiFromResources(); return Paths.get(resourceUrl.toURI()).toString(); } // 生产环境提取到临时目录 synchronized (KettleEnv.class) { if (extractedPluginPath == null) { extractedPluginPath = extractPluginsFromJar(resourceUrl); // 同时提取JNDI目录 extractJndiFromJar(); } return extractedPluginPath.toString(); } } /** * 从JAR包提取插件到临时目录(优化版) */ private static Path extractPluginsFromJar(URL jarResourceUrl) throws Exception { // 创建临时目录(带随机后缀) Path tempDir = Files.createTempDirectory("kettle-plugins-"); tempDir.toFile().deleteOnExit(); log.info("创建临时插件目录: {}", tempDir); // 解析JAR文件路径 String jarPath = jarResourceUrl.getPath().split("!")[0].replace("file:", ""); File jarFile = new File(jarPath); try (JarFile jar = new JarFile(jarFile)) { Enumeration<JarEntry> entries = jar.entries(); while (entries.hasMoreElements()) { JarEntry entry = entries.nextElement(); String entryName = entry.getName(); // 只处理插件目录下的文件 if (entryName.startsWith(PLUGIN_DIR + "/") && !entry.isDirectory()) { // 计算目标路径(去掉插件目录前缀) String relativePath = entryName.substring(PLUGIN_DIR.length() + 1); Path targetPath = tempDir.resolve(relativePath); // 创建父目录 Files.createDirectories(targetPath.getParent()); // 复制文件内容 try (InputStream in = jar.getInputStream(entry)) { Files.copy(in, targetPath, StandardCopyOption.REPLACE_EXISTING); log.debug("提取插件文件: {} -> {}", entryName, targetPath); } } } } log.info("成功提取 {} 个插件文件到临时目录", Files.list(tempDir).count()); return tempDir; } /** * 从JAR包提取JNDI配置到临时目录 */ private static void extractJndiFromJar() throws Exception { URL jndiResourceUrl = KettleUtil.class.getClassLoader().getResource(JNDI_DIR); if (jndiResourceUrl == null) { log.warn("未找到JNDI资源目录"); return; } // 创建临时目录(带随机后缀) Path tempDir = Files.createTempDirectory("kettle-jndi-"); tempDir.toFile().deleteOnExit(); log.info("创建临时JNDI目录: {}", tempDir); // 解析JAR文件路径 String jarPath = jndiResourceUrl.getPath().split("!")[0].replace("file:", ""); File jarFile = new File(jarPath); try (JarFile jar = new JarFile(jarFile)) { Enumeration<JarEntry> entries = jar.entries(); while (entries.hasMoreElements()) { JarEntry entry = entries.nextElement(); String entryName = entry.getName(); // 只处理JNDI目录下的文件 if (entryName.startsWith(JNDI_DIR + "/") && !entry.isDirectory()) { // 计算目标路径(去掉JNDI目录前缀) String relativePath = entryName.substring(JNDI_DIR.length() + 1); Path targetPath = tempDir.resolve(relativePath); // 创建父目录 Files.createDirectories(targetPath.getParent()); // 复制文件内容 try (InputStream in = jar.getInputStream(entry)) { Files.copy(in, targetPath, StandardCopyOption.REPLACE_EXISTING); log.debug("提取JNDI文件: {} -> {}", entryName, targetPath); } } } } extractedJndiPath = tempDir; log.info("成功提取JNDI配置到临时目录: {}", tempDir); } /** * 在开发环境中提取JNDI配置 */ private static void extractJndiFromResources() throws Exception { URL jndiResourceUrl = KettleUtil.class.getClassLoader().getResource(JNDI_DIR); if (jndiResourceUrl == null) { log.warn("未找到JNDI资源目录"); return; } // 开发环境直接使用资源路径 extractedJndiPath = Paths.get(jndiResourceUrl.toURI()); log.info("使用开发环境JNDI目录: {}", extractedJndiPath); } /** * 重新加载Kettle环境(包括JNDI配置) */ public static synchronized void reload() { try { // 停止所有正在运行的作业 KettleJobManager.stopAllJobs(); // 1. 关闭当前Kettle环境 KettleEnvironment.shutdown(); log.info("Kettle环境已关闭"); // 3. 重新初始化 init(); log.info("Kettle环境重新初始化完成"); } catch (Exception e) { log.error("Kettle环境重新加载失败", e); throw new RuntimeException(e); } } } // 保存配置内容(保留原始格式) @PostMapping("/saveConfig") @ResponseBody public AjaxResult saveConfig(@RequestParam String content) { try { // 直接保存原始内容,不解析为Properties KettleEnv.writeJdbcPropertiesFromString(content); // 重新加载Kettle环境 KettleEnv.reload(); return success("配置保存成功"); } catch (Exception e) { return error("保存配置失败: " + e.getMessage()); } } 保存配置时把原文件也修改 ruoyi-admin/src/main/resources/simple-jndi/jdbc.properties ruoyi-admin/target/classes/simple-jndi/jdbc.properties
最新发布
09-17
//执行本地文件作业 public boolean runKettleJob(Map<String, String> initKettleParam, String jobName) throws URISyntaxException, FileNotFoundException { KettleEnv.init(); String uuid = UUID.randomUUID().toString(); String pluginBasePath = KettleEnv.getPluginBasePath(); if (pluginBasePath == null) { throw new IllegalStateException("Kettle插件基础路径未初始化"); } // 设置JNDI路径(基于插件路径) Path jndiPath = Paths.get(pluginBasePath, "simple-jndi"); Const.JNDI_DIRECTORY = jndiPath.toString(); // 设置系统属性 System.setProperty("KETTLE_HOME", pluginBasePath); System.setProperty("org.osjava.sj.root", jndiPath.toString()); log.info("Kettle插件基础路径: {}", pluginBasePath); log.info("JNDI路径: {}", jndiPath); Path jobPath = Paths.get(jobName); log.info("JjobPath: {}", jobPath.toString()); DataSource ds = null; try { ds = (DataSource) new InitialContext().lookup("ORA_BIM"); } catch (NamingException e) { throw new RuntimeException(e); } try (Connection conn = ds.getConnection()) { log.info("JNDI Connection Successful!"); } catch (SQLException e) { throw new RuntimeException(e); } 09:49:35.706 [Thread-19] INFO c.r.k.t.KettleUtil - [init,543] - Kettle环境初始化成功 09:49:35.707 [Thread-19] INFO c.r.k.t.KettleUtil - [runKettleJob,266] - Kettle插件基础路径: D:\IDEA\java_kettle\smart-kettle\RuoYi-master\ruoyi-admin\target\classes\kettlePlugins 09:49:35.707 [Thread-19] INFO c.r.k.t.KettleUtil - [runKettleJob,267] - JNDI路径: D:\IDEA\java_kettle\smart-kettle\RuoYi-master\ruoyi-admin\target\classes\kettlePlugins\simple-jndi 09:49:35.707 [Thread-19] INFO c.r.k.t.KettleUtil - [runKettleJob,270] - JjobPath: D:\kettle\pims_etl_jndi\pims_job_jndi.kjb java.lang.NullPointerException at com.ruoyi.kettle.tools.KettleUtil.runKettleJob(KettleUtil.java:279) at com.ruoyi.kettle.service.impl.KettleJobServiceImpl.runJobRightNow(KettleJobServiceImpl.java:259) at com.ruoyi.kettle.tools.RedisStreamUtil.readGroup(RedisStreamUtil.java:251) at com.ruoyi.kettle.tools.CommandLineRunnerImpl$1.run(CommandLineRunnerImpl.java:26) D:\IDEA\java_kettle\smart-kettle\RuoYi-master\ruoyi-admin\target\classes\kettlePlugins\simple-jndi目录下jdbc.properties配置如下: SampleData/type=javax.sql.DataSource SampleData/driver=org.h2.Driver SampleData/url=jdbc:h2:file:samples/db/sampledb;IFEXISTS=TRUE SampleData/user=PENTAHO_USER SampleData/password=PASSWORD Quartz/type=javax.sql.DataSource Quartz/driver=org.hsqldb.jdbcDriver Quartz/url=jdbc:hsqldb:hsql://localhost/quartz Quartz/user=pentaho_user Quartz/password=password Hibernate/type=javax.sql.DataSource Hibernate/driver=org.hsqldb.jdbcDriver Hibernate/url=jdbc:hsqldb:hsql://localhost/hibernate Hibernate/user=hibuser Hibernate/password=password Shark/type=javax.sql.DataSource Shark/driver=org.hsqldb.jdbcDriver Shark/url=jdbc:hsqldb:hsql://localhost/shark Shark/user=sa Shark/password= PDI_Operations_Mart/type=javax.sql.DataSource PDI_Operations_Mart/driver=org.postgresql.Driver PDI_Operations_Mart/url=jdbc:postgresql://localhost:5432/hibernate?searchpath=pentaho_operations_mart PDI_Operations_Mart/user=hibuser PDI_Operations_Mart/password=password live_logging_info/type=javax.sql.DataSource live_logging_info/driver=org.postgresql.Driver live_logging_info/url=jdbc:postgresql://localhost:5432/hibernate?searchpath=pentaho_dilogs live_logging_info/user=hibuser live_logging_info/password=password ORA_BIMS/type=javax.sql.DataSource ORA_BIMS/driver=com.mysql.cj.jdbc.Driver ORA_BIMS/url=jdbc:mysql://192.168.168.217:3306/plasma_kb?useUnicode=true&characterEncoding=utf8&useSSL=false&serverTimezone=GMT%2B8 ORA_BIMS/user=plasma ORA_BIMS/password=szrouting2004plasma ORA_BIM/type=javax.sql.DataSource ORA_BIM/driver=oracle.jdbc.driver.OracleDriver ORA_BIM/url=jdbc:oracle:thin:@192.168.168.218:1521:orcl ORA_BIM/user=BIM ORA_BIM/password=szrouting2015bip 经过验证配置没有问题,是哪里出错了
06-07
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值