drl_blogs-优快云博客

原创 Vue的路由实现：hash模式和 history模式

hash模式：在浏览器中符号“#”，#以及#后面的字符称之为hash，用window.location.hash读取；特点：hash虽然在URL中，但不被包括在HTTP请求中；用来指导浏览器动作，对服务端安全无用，hash不会重加载页面。 hash 模式下，仅 hash 符号之前的内容会被包含在请求中，如 http://www.xxx.com，因此对于后端来说，即使没有做到对路由的全覆盖，也不会...

2019-10-18 20:11:45 402

原创 mysql隔离级别

事务的四种隔离级别隔离级别脏读（Dirty Read）不可重复读（NonRepeatable Read）幻读（Phantom Read）未提交读（Read uncommitted）可能可能可能已提交读（Read committed）不可能可能可能可重复读（Repeatable read）不可能不可能可能可串行化（Serializable...

2019-10-18 19:52:16 215

原创 nginx-1.16.0安装

上传 nginx-1.16.0.tar.gz 解压 tar -zxvf nginx-1.16.0.tar.gz -C /usr/local/src/ 进入解压的目录 cd /usr/local/src/nginx-1.16.0 安装插件 yum install -y gcc openssl openssl-devel pcre-devel 预编译 ./configure ...

2019-07-13 14:57:46 981

原创 MyUtil：二分查找法（Scala)

object BinarySearch { def main(args: Array[String]): Unit = { val arr = Array(1, 2, 3, 10, 100, 19999, -1998, 9, 234, 234, 9, 43) val index = Binarysearch(arr, 100) println("the found...

2019-07-13 10:54:55 229

原创 MyUtil:保存数据到Mysql

import java.sql.DriverManager import org.apache.kafka.common.TopicPartition import org.apache.spark.rdd.RDD import org.apache.spark.streaming.kafka010.OffsetRange import scala.collection.mutable objec...

2019-07-13 08:51:44 273

原创 MyUtil：时间戳转成时间字符串

import java.text.SimpleDateFormat import java.util.Date object tranTimeToString { def main(args: Array[String]): Unit = { val tm = "1502036122000" val a = tranTimeToString(tm) println...

2019-07-12 10:42:49 180

原创 MyUtil：时间字符串转时间戳

package util import java.text.SimpleDateFormat object tranTimeToLong { def main(args: Array[String]): Unit = { val tm = "20170801164432123" val a = tranTimeToLong(tm) println(a) } ...

2019-07-12 09:34:06 121

原创 MyUtil：保存数据到redis

import org.apache.spark.rdd.RDD import org.apache.spark.streaming.kafka010.OffsetRange import redis.Jpools object DataToRedis { //保存数据到redis def saveDataOffset(result1: RDD[((String,String,String...

2019-07-10 17:13:01 332

原创 flume常用配置文件

文章目录kafka-hdfs.confflume-kafka.confflume-hdfs.confflume_pull_streaming.confflume_push_streaming.conf kafka-hdfs.conf #定义agent名， source、channel、sink的名称 a1.channels = c1 a1.sinks = k1 a1.channels.c1.ty...

2019-07-09 14:24:13 689

原创安装 flink-1.8.1-bin-scala_2.11

1. 上传文件 flink-1.8.1-bin-scala_2.11.tgz 2. 解压文件 tar -zxvf flink-1.8.1-bin-scala_2.11.tgz -C /usr/local/ 3. 配置环境变量 vim /etc/profile 修改内容如下 #flink export FLINK_HOME=/usr/local/flink-1.8.1 export PAT...

2019-07-08 19:18:16 1683

原创示例：Spark Streming+Kafka整合(spark-streaming-kafka-0-10_2.11)

import org.apache.kafka.common.serialization.StringDeserializer import org.apache.log4j.{Level, Logger} import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka010.ConsumerStrategies....

2019-07-05 11:01:03 2909

原创示例：python模拟日志生成+Flume+Kafka+Spark Streaming

生成模拟数据编写 generate_log.py #coding=UTF-8 import random import time url_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...

2019-07-02 16:44:44 1047

原创【error】SparkUI端口被占用

ERROR ui.SparkUI: Failed to bind SparkUI java.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropri...

2019-07-02 14:35:00 1874 1

原创示例：Log4j日志模拟数据整合 Flume+Kafka+Spark Streaming

flume_kafka.conf agent1.sources = avro-source agent1.channels = logger-channel agent1.sinks = kafka-sink # define source agent1.sources.avro-source.type = avro agent1.sources.avro-source.bind = 0....

2019-07-01 21:22:06 296

原创示例：Spark Streming+Kafka整合 (spark-streaming-kafka-0-8_2.11)

文章目录ReceiverDirect Receiver import org.apache.log4j.{Level, Logger} import org.apache.spark.SparkConf import org.apache.spark.streaming.kafka.KafkaUtils import org.apache.spark.streaming.{Seconds, Str...

2019-07-01 17:24:20 2339

原创示例：Spark Streaming+Flume整合

文章目录pushpull push import org.apache.log4j.{Level, Logger} import org.apache.spark.SparkConf import org.apache.spark.streaming.dstream.ReceiverInputDStream import org.apache.spark.streaming.flume.{Flum...

2019-07-01 14:49:50 275

原创 Spark：【error】无法解析重载方法“agg”

错误信息：Cannot resolve overloaded method 'agg' 解决方法：导入包 import org.apache.spark.sql.functions._

2019-06-29 13:13:50 2906

原创示例：根据IP解析位置

下载公共开源项目：https://github.com/wzhe06/ipdatabase 打JAR包上传到maven示例：https://blog.youkuaiyun.com/drl_blogs/article/details/94011565 引入jar包和依赖 <dependency> <groupId>com.ggstar</groupI...

2019-06-28 17:19:26 625

原创示例：redis输入/输出

创建连接池：https://blog.youkuaiyun.com/drl_blogs/article/details/93979067 写入redis //df为已经处理好的DataFrame df.foreachPartition(filter=>{ val jedis = Jpools.getJedis filter.foreach(tp=>{ ...

2019-06-28 12:22:23 354

原创 MyUtil：创建jedis连接池

package MyUtil import org.apache.commons.pool2.impl.GenericObjectPoolConfig import redis.clients.jedis.{Jedis, JedisPool} /** * 创建jedis连接池 */ object Jpools { //获取连接对象 private val config = n...

2019-06-28 11:19:34 201

原创 MyUtil：将[dd/MMM/yyyy:HH:mm:ss Z]类型时间转换为yyyy-MM-dd hh:mm:ss格式

package MyUtil import java.util.{Date, Locale} import org.apache.commons.lang3.time.FastDateFormat object DateUtils { //日期时间格式 [10/Nov/2016:00:01:02 +0800] val YYYYMMDDHHMM_TIME_FORMAT = FastDa...

2019-06-27 23:50:57 1025

原创 flume-1.9.0安装

上传 apache-flume-1.9.0-bin.tar.gz 解压 tar -zxvf apache-flume-1.9.0-bin.tar.gz -C /usr/local/ 配置环境变量 vim /etc/profile 添加内容 export FLUME_HOME=/usr/local/apache-flume-1.9.0-bin export PATH=$PATH:$FL...

2019-06-26 15:14:29 1105

原创示例：Kafka Stream 数据清洗

接受数据后的处理逻辑 import org.apache.kafka.streams.processor.Processor; import org.apache.kafka.streams.processor.ProcessorContext; public class LogProcessor implements Processor<byte[], byte[]> { p...

2019-06-25 16:07:44 1201

原创示例：kafka拦截器

拦截内容在内容前加时间戳 import java.util.Map; import org.apache.kafka.clients.producer.ProducerInterceptor; import org.apache.kafka.clients.producer.ProducerRecord; import org.apache.kafka.clients.producer.Re...

2019-06-25 14:58:48 344

原创示例：统计IP地址对应的省份，并把结果存入到mysql

数据 ip.txt access.log import java.io.{BufferedReader, FileInputStream, InputStreamReader} import java.sql.{Connection, DriverManager, PreparedStatement} import org.apache.spark.{SparkConf, SparkCont...

2019-06-25 10:20:58 936 3

原创 kafka_2.11-0.11.0.0 安装配置

1.上传kafka_2.11-0.11.0.0.tgz 2.解压 tar -zxvf kafka_2.11-0.11.0.0.tgz -C /usr/local/ 3.修改server.properties vim server.properties 修改内容如 #broker的全局唯一编号，不能重复 broker.id=0 #删除topic功能使能 delete.topic.enable=t...

2019-06-24 10:32:18 808

原创示例：Spark SQL自定义函数（UDF/UDAF）

文章目录UDF函数UDAF函数 UDF函数 scala> val df=spark.read.json("people.json") df: org.apache.spark.sql.DataFrame = [age: bigint, name: string] scala> df.show +---+------+ |age| name| +---+------+ | 30| ...

2019-06-20 17:12:00 1587

原创 Spark：【error】DataFrames转DataSet失败

Error:(45, 63) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for seria...

2019-06-20 17:01:13 328

原创 Spark ：【error】System memory 259522560 must be at least 471859200

java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.。。。。。。。...

2019-06-20 15:36:29 466

原创 Spark:常用JAR包集成（pom.xml）

<?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ma...

2019-06-19 23:17:11 713

原创 Spark SQL：RDD、DataFrames、DataSet之间的转换

文章目录RDD转DataFramesRDD转DataSetDataFrame/Dataset转RDDDataFrame转DatasetDataset转DataFrame people.txt Michael,29 Andy,30 Justin,19 RDD转DataFrames scala> val rdd=sc.textFile("people.txt") rdd: org.apache...

2019-06-19 16:55:28 279

原创 redis:编译时报错 /bin/sh: cc: command not found

[root@hadoop01 redis-3.0.0]# make cd src && make all make[1]: Entering directory `/usr/local/redis-3.0.0/src' rm -rf redis-server redis-sentinel redis-cli redis-benchmark redis-check-dump redi...

2019-06-19 16:54:52 1891

原创 Spark Core:数据输入输出

文本文件输入输出读取文本文件 scala> sc.textFile("./wc.txt") res4: org.apache.spark.rdd.RDD[String] = ./wc.txt MapPartitionsRDD[5] at textFile at <console>:25 保存文本文件 scala> res4.saveAsTextFile("./test"...

2019-06-19 16:54:20 503

原创 Spark Core:RDD编程Action

Action目录reduce(f: (T, T) => T): Tcollect(): Array[T]count(): Longfirst(): Ttake(num: Int): Array[T]takeOrdered(num: Int)(implicit ord: Ordering[T])aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T)...

2019-06-19 16:53:59 196

原创 Spark Core:RDD编程Transformation

文章目录创建RDD操作map[U: ClassTag](f: T => U): RDD[U]filter(f: T => Boolean): RDD[T]flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U]mapPartition[U: ClassTag]( f: Iterator[T] => Iterator[U...

2019-06-19 16:53:34 713

原创 Spark Core:创建RDD

#方式一：把数据并行化分片到节点 sc.parallelize(Array(1,2,3,4)) #方式二：把数据并行化分片到节点 sc.makeRDD(Array(1,2,3)) #方式三：可以指定RDD的存放位置 #创建一个list集合 val list1=List((1,List("Hello","Word","spark")),(2,List("at","as"))) #将List集合放入R...

2019-06-17 16:35:16 198

原创 Spark：【error】远程调试 root:supergroup:drwxr-xr-x

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxx, access=WRITE, inode="/test/out/_temporary/0":root:supergroup:drwxr-xr-x .....................

2019-06-15 16:27:46 869

原创 spark-2.2.2-bin-hadoop2.7 HA 配置

安装spark-2.2.2-bin-hadoop2.7：https://blog.youkuaiyun.com/drl_blogs/article/details/91948394 1.编辑主节点conf/spark-env.sh export JAVA_HOME=/usr/local/jdk1.8.0_211 # export SPARK_MASTER_HOST=hadoop01 # export ...

2019-06-14 11:08:28 327

原创 spark-2.2.2-bin-hadoop2.7 安装

1.上传spark-2.2.2-bin-hadoop2.7.tgz 2.解压文件 tar -zxvf spark-2.2.2-bin-hadoop2.7.tgz -C /usr/local/ 3.进入conf/下把spark-env.sh.template改为spark-env.sh cd /usr/local/spark-2.2.2-bin-hadoop2.7/conf/ mv s...

2019-06-14 09:23:37 3516

原创示例：Scala读取xml文件

test.xml内容如下 <?xml version="1.0"?> <Tests> <Test TestId="0001" TestType="CMD"> <Name>Convert number to string</Name> <CommandLine>Examp1.EXE<...

2019-06-13 15:10:47 2167 1

空空如也

空空如也