flume HttpSource使用

最新推荐文章于 2024-08-06 20:30:52 发布

SW_LCC

最新推荐文章于 2024-08-06 20:30:52 发布

阅读量999

点赞数

CC 4.0 BY-SA版权

分类专栏： flume 文章标签：大数据 flume java

本文链接：https://blog.youkuaiyun.com/SW_LCC/article/details/100026665

flume 专栏收录该内容

12 篇文章

订阅专栏

本文深入解析Flume HttpSource组件，介绍其如何接收GET和POST请求，并通过HttpHandler转换为Event，适用于游戏客户端等无法部署SDK的日志收集场景。文章详细展示了HttpSource的配置参数、二次开发示例及异常处理。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

HttpSource

httpsource允许是flume提供的能接收get和post的http请求，http请求会被HttpHandler接口实现类处理，从而将http请求转换为event从而提交到channel。
http适用于一些无法通过部署SDK接收日志的场景，例如游戏客户端。

HttpSource配置使用

HttpSource基本配置参数

配置参数	默认值	描述
type	-	http (org.apache.fluem.source.httpSource）或HttpSource
bind	0.0.0.0	绑定的IP地址或主机名
port	-	绑定的端口号
enableSSL	false	是否启用SSL
keystore	-	keystore文件的路径
keystorePassword	-	进入keystore的密码
handler	JSONHandler	HttpSource使用的处理程序类

配置文件示例:

#source
a1.sources = r1
a1.channels = c1
a1.sources.r1.type = http
a1.sources.r1.bind=0.0.0.0
a1.sources.r1.port = 5140
a1.sources.r1.channels = c1
a1.sources.r1.hadler=HTTPSourceXMLHandler
#channel
a1.channels.c1.type=memory
a1.channels.c1.capacity=10000
a1.channels.c1.transactionCapacity=1000
#sink
a1.sinks.k1.type=logger
a1.sinks.k1.channel=c1

HttpHandler二次开发

实现HttpSourceHandler接口，实现getEvents方法，接收HttpServletRequest参数，返回生成的List集合

常见示例如下:

public class HTTPSourceXMLHandler implements HTTPSourceHandler {
    private final String ROOT = "events";
    private final String EVENT_TAG = "event";
    private final String HEADERS_TAG = "headers";
    private final String BODY_TAG = "body";

    private final String CONF_INSERT_TIMESTAMP = "insertTimestamp";
    private final String TIMESTAMP_HEADER = "timestamp";
    private final DocumentBuilderFactory documentBuilderFactory
            = DocumentBuilderFactory.newInstance();

    // Document builders are not thread-safe.
    // So make sure we have one for each thread.
    private final ThreadLocal<DocumentBuilder> docBuilder
            = new ThreadLocal<DocumentBuilder>();

    private boolean insertTimestamp;
    private static final Logger LOG = LoggerFactory.getLogger(HTTPSourceXMLHandler.class);


    public List<Event> getEvents(HttpServletRequest httpServletRequest) throws HTTPBadRequestException, Exception {
        if (docBuilder.get() == null) {
            docBuilder.set(documentBuilderFactory.newDocumentBuilder());
        }
        Document doc;
        final List<Event> events;
        try {
            doc = docBuilder.get().parse(httpServletRequest.getInputStream());            
            Element root = doc.getDocumentElement();        

            root.normalize();
            // Verify that the root element is "events"
            Preconditions.checkState(
                    ROOT.equalsIgnoreCase(root.getTagName()));

            NodeList nodes = root.getElementsByTagName(EVENT_TAG);
            LOG.info("get nodes={}",nodes);

            int eventCount = nodes.getLength();
            events = new ArrayList<Event>(eventCount);
            for (int i = 0; i < eventCount; i++) {
                Element event = (Element) nodes.item(i);
                // Get all headers. If there are multiple header sections,
                // combine them.
                NodeList headerNodes
                        = event.getElementsByTagName(HEADERS_TAG);
                Map<String, String> eventHeaders
                        = new HashMap<String, String>();
                for (int j = 0; j < headerNodes.getLength(); j++) {
                    Node headerNode = headerNodes.item(j);
                    NodeList headers = headerNode.getChildNodes();
                    for (int k = 0; k < headers.getLength(); k++) {
                        Node header = headers.item(k);

                        // Read only element nodes```
                        if (header.getNodeType() != Node.ELEMENT_NODE) {
                            continue;
                        }
                        // Make sure a header is inserted only once,
                        // else the event is malformed
                        Preconditions.checkState(
                                !eventHeaders.containsKey(header.getNodeName()),
                                "Header expected only once " + header.getNodeName());
                        eventHeaders.put(
                                header.getNodeName(), header.getTextContent());
                    }
                }
                Node body = event.getElementsByTagName(BODY_TAG).item(0);
                if (insertTimestamp) {
                    eventHeaders.put(TIMESTAMP_HEADER, String.valueOf(System
                            .currentTimeMillis()));
                }
                events.add(EventBuilder.withBody(
                        body.getTextContent().getBytes(
                                httpServletRequest.getCharacterEncoding()),
                        eventHeaders));
            }
        } catch (SAXException ex) {
            throw new HTTPBadRequestException(
                    "Request could not be parsed into valid XML", ex);
        } catch (Exception ex) {
            throw new HTTPBadRequestException(
                    "Request is not in expected format. " +
                            "Please refer documentation for expected format.", ex);
        }
        return events;
    }

    public void configure(Context context) {
        insertTimestamp = context.getBoolean(CONF_INSERT_TIMESTAMP,
                false);
    }
}