flume Channel选择器

简介

channel处理器用于决定source接收的时间写入哪个channel，由其通知channel，并将时间写入。

选择器类型

flume自带两种选择器:
1.relicating(复制选择器）,默认使用
2.multiplexing(多路复用选择器)

1.复制选择器

复制选择器会复制每个source接收的事件，将事件复制并分发到所有channel，可通过配置参数，控制分发的channel。

配置参数

参数	描述
type	replicating
optional	channel名（即channel可选，失败时不会再次发送）

配置示例:

agent.sources.s1.type=avro
agent.sources.s1.channels = c1 c2 c3
agnet.sources.s1.selector.type=relicating
agent.sources.s1.selector.optional= c3

按上面配置，当source写c3失败时，不会抛出ChannelException异常，因此c3也不会导致source发送重试。

源码分析:

/**
 * Replicating channel selector. This selector allows the event to be placed
 * in all the channels that the source is configured with.
 */
public class ReplicatingChannelSelector extends org.apache.flume.channel.AbstractChannelSelector {

  /**
   * Configuration to set a subset of the channels as optional.
   */
  public static final String CONFIG_OPTIONAL = "optional";
  List<Channel> requiredChannels = null;
  List<Channel> optionalChannels = new ArrayList<Channel>();

  @Override
  //返回所有除option的channel列表
  public List<Channel> getRequiredChannels(Event event) {
    /*
     * Seems like there are lot of components within flume that do not call
     * configure method. It is conceiveable that custom component tests too
     * do that. So in that case, revert to old behavior.
     */
    if (requiredChannels == null) {
      return getAllChannels();
    }
    return requiredChannels;
  }

  @Override
  //返回所有option channel列表
  public List<Channel> getOptionalChannels(Event event) {
    return optionalChannels;
  }

  @Override
  public void configure(Context context) {
    String optionalList = context.getString(CONFIG_OPTIONAL);
    requiredChannels = new ArrayList<Channel>(getAllChannels());
    Map<String, Channel> channelNameMap = getChannelNameMap();
    if (optionalList != null && !optionalList.isEmpty()) {
      for (String optional : optionalList.split("\\s+")) {
        Channel optionalChannel = channelNameMap.get(optional);
        //如channel在channels后配置了，也在option配置了，则应从去除，只作为option channel
        requiredChannels.remove(optionalChannel);
        if (!optionalChannels.contains(optionalChannel)) {
          optionalChannels.add(optionalChannel);
        }
      }
    }
  }
}

2.多路复用选择器

多路复用选择器常通过报头的某个值来选择写入哪一个channel。

配置参数

参数	描述
type	multiplexing
header	报头
mapping.<header.value>	报头映射列表，由报头的值匹配
optional.<header.value>	与mapping一样，但写入失败会被忽略

配置示例：

agent.sources.s1.type=avro
agent.sources.s1.channels = c1 c2 c3 c4 c5
agnet.sources.s1.selector.type= multiplexing
agent.sources.s1.selector.header= priority
agent.sources.s1.selector.mapping.1 = c1 c2
agent.sources.s1.selector.mapping.2 = c2
agent.sources.s1.selector.optional.1= c3 
agent.sources.s1.selector.optional.2= c4 
agent.sources.s1.selector.optional.3= c4
agent.sources.s1.selector.default= c5

如按上配置，选择器会根据header里面的键priority（键）进行选择。
当值为1时，会写入c1,c2,c3，若写入c3出错，不会重试。
当值为2时，会写入c2和c4，写入c4，出错不会重试。
当值为3时，只写入c4，c5，应为没有mapping映射c3, 故会写如c5,且c4出错不重试。
当值不为1，2，3时，则会写入c5

源码阅读:

public class MultiplexingChannelSelector extends org.apache.flume.channel.AbstractChannelSelector {

  public static final String CONFIG_MULTIPLEX_HEADER_NAME = "header";
  public static final String DEFAULT_MULTIPLEX_HEADER ="flume.selector.header";
  public static final String CONFIG_PREFIX_MAPPING = "mapping.";
  public static final String CONFIG_DEFAULT_CHANNEL = "default";
  public static final String CONFIG_PREFIX_OPTIONAL = "optional";

  @SuppressWarnings("unused")
  private static final Logger LOG = LoggerFactory.getLogger(MultiplexingChannelSelector.class);

  private static final List<Channel> EMPTY_LIST =
      Collections.emptyList();

  //header参数值
  private String headerName;

  //channel映射表
  private Map<String, List<Channel>> channelMapping;
  //option channel映射表
  private Map<String, List<Channel>> optionalChannels;
  //default channel集合
  private List<Channel> defaultChannels;

  @Override
  public List<Channel> getRequiredChannels(Event event) {
    String headerValue = event.getHeaders().get(headerName);
    //header参数值不存在返回default的channel
    if (headerValue == null || headerValue.trim().length() == 0) {
      return defaultChannels;
    }

    //由header的值获取channel
    List<Channel> channels = channelMapping.get(headerValue);

    //header参数值
    if (channels == null) {
      channels = defaultChannels;
    }

    return channels;
  }

  @Override
  public List<Channel> getOptionalChannels(Event event) {
    String hdr = event.getHeaders().get(headerName);
    List<Channel> channels = optionalChannels.get(hdr);

    if (channels == null) {
      channels = EMPTY_LIST;
    }
    return channels;
  }

  @Override
  public void configure(Context context) {
    this.headerName = context.getString(CONFIG_MULTIPLEX_HEADER_NAME,
        DEFAULT_MULTIPLEX_HEADER);
    //获取所有的channel，包括option和mapping，default
    Map<String, Channel> channelNameMap = getChannelNameMap();

    //default Channels
    defaultChannels = getChannelListFromNames(
        context.getString(CONFIG_DEFAULT_CHANNEL), channelNameMap);

    Map<String, String> mapConfig =
        context.getSubProperties(CONFIG_PREFIX_MAPPING);

    // mapping映射的channel
    channelMapping = new HashMap<String, List<Channel>>();

    for (String headerValue : mapConfig.keySet()) {
      List<Channel> configuredChannels = getChannelListFromNames(
          mapConfig.get(headerValue),
          channelNameMap);

      //This should not go to default channel(s)
      //because this seems to be a bad way to configure.
      if (configuredChannels.size() == 0) {
        throw new FlumeException("No channel configured for when "
            + "header value is: " + headerValue);
      }
      //同样的值被配置了两次
      if (channelMapping.put(headerValue, configuredChannels) != null) {
        throw new FlumeException("Selector channel configured twice");
      }
    }
    //If no mapping is configured, it is ok.
    //All events will go to the default channel(s).
    //如果header value没有mapping或者optional任何channel，则写入到default
    Map<String, String> optionalChannelsMapping = context.getSubProperties(CONFIG_PREFIX_OPTIONAL + ".");

    optionalChannels = new HashMap<String, List<Channel>>();

    for (String hdr : optionalChannelsMapping.keySet()) {
      List<Channel> confChannels = getChannelListFromNames(
              optionalChannelsMapping.get(hdr), channelNameMap);
      if (confChannels.isEmpty()) {
        confChannels = EMPTY_LIST;
      }
      //Remove channels from optional channels, which are already
      //configured to be required channels.
      //没有mapping只有option则要加上default的channel
      List<Channel> reqdChannels = channelMapping.get(hdr);
      //Check if there are required channels, else defaults to default channels
      if (reqdChannels == null || reqdChannels.isEmpty()) {
        reqdChannels = defaultChannels;
      }
      for (Channel c : reqdChannels) {
        if (confChannels.contains(c)) {
          confChannels.remove(c);
        }
      }

      if (optionalChannels.put(hdr, confChannels) != null) {
        throw new FlumeException("Selector channel configured twice");
      }
    }

  }

}