kettle内存溢出后执行垃圾回收

kettle运行时会报内存溢出,猜测是kettle运行太久后内存堆积,而kettle没有自动GC,后面又有占用内存的任务启动时就会报错,错误信息如图:

在这里插入图片描述

本人的解决办法是:增加一个转换,计算当前已使用内存是否达到阈值,达到阈值就使用java代码调用GC

运行效果如下:

在这里插入图片描述

在表输入中设置内存使用率达到多少G时执行GC

在这里插入图片描述

在java代码中获取已用内存和总内存,当内存达到表输入的阈值时触发GC

在这里插入图片描述

java代码如下:

import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.lang.management.MemoryUsage;

public boolean processRow(StepMetaInterface smi, StepDataInterface sdi) throws KettleException {
	if (first) {
		first = false;

    /* TODO: Your code here. (Using info fields)

    FieldHelper infoField = get(Fields.Info, "info_field_name");

    RowSet infoStream = findInfoRowSet("info_stream_tag");

    Object[] infoRow = null;

    int infoRowCount = 0;

    // Read all rows from info step before calling getRow() method, which returns first row from any
    // input rowset. As rowMeta for info and input steps varies getRow() can lead to errors.
    while((infoRow = getRowFrom(infoStream)) != null){

      // do something with info data
      infoRowCount++;
    }
    */
	}

	MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();

	MemoryUsage heapMemoryUsage = memoryMXBean.getHeapMemoryUsage();

	// 获取已使用内存大小(bytes)
	long usedMemory = heapMemoryUsage.getUsed();

	// 获取已分配内存大小(bytes)
	long committedMemory = heapMemoryUsage.getCommitted();

	// 获取最大内存大小(bytes)
	long maxMemory = heapMemoryUsage.getMax();

	Object[] r = getRow();

	if (r == null) {
		setOutputDone();
		return false;
	}

	Long limitMemory = Long.parseLong(get(Fields.In, "limitMemory").getString(r));
	logBasic("kettle已使用内存:"+usedMemory/(1024 * 1024)+"M");
	logBasic("kettle已分配内存:"+committedMemory/(1024 * 1024)+"M");
	logBasic("kettle最大内存:"+maxMemory/(1024 * 1024)+"M");
	if (usedMemory > (limitMemory * 1024 * 1024 * 1024)) {
		logBasic("垃圾回收,走GC了");
		System.gc();
	}

  // It is always safest to call createOutputRow() to ensure that your output row's Object[] is large
  // enough to handle any new fields you are creating in this step.
	r = createOutputRow(r, data.outputRowMeta.size());

  /* TODO: Your code here. (See Sample)

  // Get the value from an input field
  String foobar = get(Fields.In, "a_fieldname").getString(r);

  foobar += "bar";
    
  // Set a value in a new output field
  get(Fields.Out, "output_fieldname").setValue(r, foobar);

  */
  // Send the row on to the next step.
  putRow(data.outputRowMeta, r);

  return true;
}

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值