logstash读取日志文件到odps（maxcompute）

1.首先，了解logstash的配置文件构成。

原文链接：https://blog.youkuaiyun.com/sxf_123456/article/details/77773953

logstash 条件判断语句
使用条件来决定filter和output处理特定的事件。logstash条件类似于编程语言。条件支持if、else if、else语句，可以嵌套。 
比较操作有： 
相等: ==, !=, <, >, <=, >= 
正则: =~(匹配正则), !~(不匹配正则) 
包含: in(包含), not in(不包含) 
布尔操作： 
and(与), or(或), nand(非与), xor(非或) 
一元运算符： 
!(取反) 
()(复合表达式), !()(对复合表达式结果取反) 

2.下载有logstash-output-maxcompute插件的Logstash实例。解压安装到/usr/local/logstash-7.8.0

https://odps-repo.oss-cn-hangzhou.aliyuncs.com/streaming-tunnel/logstash-7.8.0-with-mc-output.tar.gz?spm=a2c4g.11186623.2.13.329b63c1xEwGnN&file=logstash-7.8.0-with-mc-output.tar.gz

3./usr/local/logstash-7.8.0/config/logstash-odps.conf

input {
    file {
        path => "/opt/cloudflare-cn/logs/*.log"
        type => "cloudflare-cn-log"
        start_position => "beginning"
        codec => json   #转换成json
        }
}
filter {
      if [ClientRequestHost] !~ "xxxx\..*\.xxxxxxx.cn" {   #正则匹配，不匹配的丢弃这条数据,(其中\.匹配.)
        drop {}
      }
    }

output {
        maxctunnel{
                aliyun_access_id => "xxxxxxxx"
                aliyun_access_key => "xxxxxxxxxxxxxxxx"
                aliyun_mc_endpoint => "http://service.cn-hangzhou.maxcompute.aliyun.com/api"
                project => "xxxxxx"
                table => "xxxxxxxx"
                partition => "pt=$<@timestamp.strftime('%F')>"   #@timestamp为logstash自动生成的时间戳，当成分区
                #下面这些字段与转成json里的字段名对应相同，odps的表里必须存在所有字段，顺序就是下面的顺序
                value_fields => ["CacheCacheStatus", "CacheResponseBytes", "CacheResponseStatus", "CacheTieredFill", "ClientASN", "ClientCountry", "ClientDeviceType", "ClientIP", "ClientIPClass", "ClientRequestBytes", "ClientRequestHost", "ClientRequestMethod", "ClientRequestPath", "ClientRequestProtocol", "ClientRequestURI", "ClientRequestUserAgent", "ClientSSLCipher", "ClientSSLProtocol", "ClientSrcPort", "EdgeColoCode", "EdgeColoID", "EdgeEndTimestamp", "EdgePathingOp", "EdgePathingSrc", "EdgePathingStatus", "EdgeRateLimitID", "EdgeRequestHost", "EdgeResponseBytes", "EdgeResponseCompressionRatio", "EdgeResponseContentType", "EdgeResponseStatus", "EdgeServerIP", "EdgeStartTimestamp", "FirewallMatchesActions", "FirewallMatchesRuleIDs", "FirewallMatchesSources", "OriginIP", "OriginResponseBytes", "OriginResponseStatus", "OriginResponseTime", "OriginSSLProtocol", "ParentRayID", "RayID", "SecurityLevel", "WAFAction", "WAFFlags", "WAFProfile", "WorkerCPUTime", "WorkerStatus", "WorkerSubrequest", "WorkerSubrequestCount", "ZoneID"]
}
}