qq_23159975的博客

需求: 原本的日志格式183.136.128.154 - - [30/Jul/2016:10:56:24 +0800] "GET http://static.tx.wmpyol.com/play/play.html HTTP/1.1" 200 651 "-" "Go-http-client/1.1" Hit "C/200" Static "max-age=60" 0.115 59.49.8

spark日志检查–将数据写入到数据中02首先来看看链接MySQL的操作 引入的包是import java.sql.{DriverManager, PreparedStatement, Connection} var conn: Connection = null var ps: PreparedStatement = null val sql = "INSERT INTO

package sparkSQLimport org.apache.spark.sql.{DataFrame, SparkSession}/** * Created by sicong on 2017/3/9. */object sparkKodo {// def main(args: Array[String]): Unit = { val spark = SparkS

package hadoopimport java.security.MessageDigestimport java.text.SimpleDateFormatimport IPInfo.IPimport org.apache.spark.rdd.RDDimport org.apache.spark.sql.{Dataset, SQLContext, SparkSession}import

ip库的信息在这里下载 http://www.ipip.net/download.html 182.146.100.97 - 3 [03/Jan/2017:23:30:01 +0800] "GET http://7xna64.com2.z0.glb.qiniucdn.com/Fq9M_Gn0RRWy9eprb0T0CAdrybv3.jpg?imageView2/2/w/1080/h/1920&e=1