Log4j直接发送数据到Flume + Kafka (方式二)

该文是接着上篇 Log4j直接发送数据到Flume + Kafka (方式一) 所提到的问题进行展开记录的.

主要方式就是通过继承log4j的Appenders, 采用flume rpc动态连接flume服务去发送logs信息.

• flume sdk包:

<dependency>
 <groupId>org.apache.flume</groupId>
 <artifactId>flume-ng-sdk</artifactId>
 <version>1.7.0</version>
</dependency>

• 动态连接flume的代码:

public class FindAsyncLog4j2Appender extends AbstractAppender {

    //dosomething...

    private void connect(){

        Properties props = new Properties();

        if(isSingleSink(hosts)) {
            props.setProperty(RpcClientConfigurationConstants.CONFIG_HOSTS, "h1");
            props.setProperty(RpcClientConfigurationConstants.CONFIG_HOSTS_PREFIX + "h1", hosts);
            props.setProperty(RpcClientConfigurationConstants.CONFIG_CONNECT_TIMEOUT, String.valueOf(timeout));
            props.setProperty(RpcClientConfigurationConstants.CONFIG_REQUEST_TIMEOUT, String.valueOf(timeout));
        } else {
            props = getProperties(hosts, timeout);
        }

        try {
            rpcClient = RpcClientFactory.getInstance(props);
            if(!isStarted()){
                start();
            }
        } catch (FlumeException e) {
            String errormsg = "RPC client creation failed! " + e.getMessage();
            LOGGER.error(errormsg);
            throw e;
        }
    }
    /**
     * Create an AsyncAppender.
     *
     * @param blocking         True if the Appender should wait when the queue is full. The default is true.
     * @param shutdownTimeout  How many milliseconds the Appender should wait to flush outstanding log events
     *                         in the queue on shutdown. The default is zero which means to wait forever.
     * @param size             The size of the event queue. The default is 128.
     * @param name             The name of the Appender.
     * @param filter           The Filter or null.
     * @return The AsyncAppender.
     */
    @PluginFactory
    public static FindAsyncLog4j2Appender createAppender(
            @PluginAttribute(value = "blocking", defaultBoolean = true) boolean blocking,
            @PluginAttribute(value = "shutdownTimeout") long shutdownTimeout,
            @PluginAttribute(value = "bufferSize") int size,
            @PluginAttribute("name") final String name,
            @PluginAttribute("hosts") String hosts,
            @PluginAttribute(value = "timeout") Long timeout,
            @PluginElement("Filter") Filter filter,
            @PluginAttribute("application") String application,

            @PluginElement("Layout") Layout<? extends Serializable> layout) {

        if (name == null ) {
            LOGGER.error("No name provided for FindAsyncLog4j2Appender");
            return null;
        }
        if (hosts == null ) {
            LOGGER.error("No host provided for FindAsyncLog4j2Appender");
            return null;
        }

        if (application == null) {
            LOGGER.error("No application provided for FindAsyncLog4j2Appender");
            return null;
        }

        if (layout == null) {
            layout = PatternLayout.createDefaultLayout();
        }

        return new FindAsyncLog4j2Appender(name, filter, layout, size, blocking,
                shutdownTimeout, hosts, timeout, application);
    }

   //dosomething...

}

 

• 添加appender

业务程序中初始化FindAsyncLog4j2Appender, 并addAppender到log4j 配置中去.

FindAsyncLog4j2Appender appender = FindAsyncLog4j2Appender
                .createAppender( false, 0, 1024*10, appenderName, flumeHost,
                         3000L, null,"testApp", layout);
        appender.start();
        config.addAppender(appender);
        AppenderRef ref = AppenderRef.createAppenderRef(appenderName, null, null);
        AppenderRef[] refs = new AppenderRef[]{ref};

        LoggerConfig loggerConfig = LoggerConfig.createLogger(false, Level.INFO, loggerName,
                "true", refs, null, config, null);
        loggerConfig.addAppender(appender, null, null);
        config.addLogger(loggerName, loggerConfig);

 

启动业务程序, 在kafka 消费topic中查看结果. 启动步骤请查看上一篇章:Log4j直接发送数据到Flume + Kafka (方式一)

如果手动停止flume服务, 不影响业务程序正常工作, 再次启动flume服务后, 程序会自动重新连接到flume 并发送logs, 如下图:

 

该业务程序demo代码请参看 https://github.com/spring410/springbootlog4jflume