LogQuery

最新推荐文章于 2021-06-09 18:40:26 发布

润砾成珠

最新推荐文章于 2021-06-09 18:40:26 发布

阅读量886

点赞数

分类专栏： spark

本文链接：https://blog.youkuaiyun.com/u011569805/article/details/37879135

版权

spark 专栏收录该内容

14 篇文章

订阅专栏

1. 正则表达式，匹配下面string

val apacheLogRegex =
      """^([\d.]+) (\S+) (\S+) \[([\w\d:/]+\s[+\-]\d{4})\] "(.+?)" (\d{3}) ([\d\-]+) "([^"]+)" "([^"]+)".*""".r

"""10.10.10.10 - "FRED" [18/Jan/2013:17:56:07 +1100] "GET http://images.com/2013/Generic.jpg
      | HTTP/1.1" 304 315 "http://referall.com/" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;
      | GTB7.4; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; .NET CLR
      | 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR
      | 3.5.30729; Release=ARP)" "UD-1" - "image/jpeg" "whatever" 0.350 "-" - "" 265 923 934 ""
      | 62.24.11.25 images.com 1358492167 - Whatup""".stripMargin.lines.mkString

2.匹配解析，并把值（group）对应取出

def extractKey(line: String): (String, String, String) = {
      apacheLogRegex.findFirstIn(line) match {
        case Some(apacheLogRegex(ip, _, user, dateTime, query, status, bytes, referer, ua)) =>
          if (user != "\"-\"") (ip, user, query)
          else (null, null, null)
        case _ => (null, null, null)
      }
    }

def extractStats(line: String): Stats = {
      apacheLogRegex.findFirstIn(line) match {
        case Some(apacheLogRegex(ip, _, user, dateTime, query, status, bytes, referer, ua)) =>
          new Stats(1, bytes.toInt)
        case _ => new Stats(1, 0)
      }
    }

3. key 为 ip,user,query； value 为 Stats,唯一需要注意的是 Stats Class需要继承 scala.Serializable序列化,

完成的功能为计算 ip,user,query==>total bytes 再遍历输出（个人觉得 case(user,query)有误导）；

 dataSet.map(line => (extractKey(line), extractStats(line)))
      .reduceByKey((a, b) => a.merge(b))
      .collect().foreach{
        case (user, query) => println("%s\t%s".format(user, query))}
  }

class Stats(val count: Int, val numBytes: Int) extends Serializable {
      def merge(other: Stats) = new Stats(count + other.count, numBytes + other.numBytes)
      override def toString = "bytes=%s\tn=%s".format(numBytes, count)
    }