drl_blogs的博客

1.上传spark-2.2.2-bin-hadoop2.7.tgz2.解压文件tar -zxvf spark-2.2.2-bin-hadoop2.7.tgz -C /usr/local/3.进入conf/下把spark-env.sh.template改为spark-env.shcd /usr/local/spark-2.2.2-bin-hadoop2.7/conf/mv s...

文章目录pushpullpushimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.dstream.ReceiverInputDStreamimport org.apache.spark.streaming.flume.{Flum...

文章目录ReceiverDirectReceiverimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spark.streaming.{Seconds, Str...

flume_kafka.confagent1.sources = avro-sourceagent1.channels = logger-channelagent1.sinks = kafka-sink# define sourceagent1.sources.avro-source.type = avroagent1.sources.avro-source.bind = 0....

ERROR ui.SparkUI: Failed to bind SparkUIjava.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropri...

生成模拟数据编写 generate_log.py#coding=UTF-8import randomimport timeurl_paths=[ "class/112.html", "class/128.html", "class/145.html", "class/130.html", "class/146.html", "cla...

错误信息：Cannot resolve overloaded method 'agg'解决方法：导入包import org.apache.spark.sql.functions._

import org.apache.kafka.common.serialization.StringDeserializerimport org.apache.log4j.{Level, Logger}import org.apache.spark.SparkConfimport org.apache.spark.streaming.kafka010.ConsumerStrategies....

#方式一：把数据并行化分片到节点sc.parallelize(Array(1,2,3,4))#方式二：把数据并行化分片到节点sc.makeRDD(Array(1,2,3))#方式三：可以指定RDD的存放位置#创建一个list集合val list1=List((1,List("Hello","Word","spark")),(2,List("at","as")))#将List集合放入R...

文章目录创建RDD操作map[U: ClassTag](f: T => U): RDD[U]filter(f: T => Boolean): RDD[T]flatMap[U: ClassTag](f: T => TraversableOnce[U]): RDD[U]mapPartition[U: ClassTag]( f: Iterator[T] => Iterator[U...

安装spark-2.2.2-bin-hadoop2.7：https://blog.youkuaiyun.com/drl_blogs/article/details/919483941.编辑 主节点conf/spark-env.shexport JAVA_HOME=/usr/local/jdk1.8.0_211# export SPARK_MASTER_HOST=hadoop01# export ...

Action目录reduce(f: (T, T) => T): Tcollect(): Array[T]count(): Longfirst(): Ttake(num: Int): Array[T]takeOrdered(num: Int)(implicit ord: Ordering[T])aggregate[U: ClassTag](zeroValue: U)(seqOp: (U, T)...

文本文件输入输出读取文本文件scala> sc.textFile("./wc.txt")res4: org.apache.spark.rdd.RDD[String] = ./wc.txt MapPartitionsRDD[5] at textFile at <console>:25保存文本文件scala> res4.saveAsTextFile("./test"...

文章目录RDD转DataFramesRDD转DataSetDataFrame/Dataset转RDDDataFrame转DatasetDataset转DataFramepeople.txtMichael,29Andy,30Justin,19RDD转DataFramesscala> val rdd=sc.textFile("people.txt")rdd: org.apache...

<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://ma...

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=xxxx, access=WRITE, inode="/test/out/_temporary/0":root:supergroup:drwxr-xr-x.....................

java.lang.IllegalArgumentException: System memory 259522560 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.。。。。。。。...

Error:(45, 63) Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for seria...

文章目录UDF函数UDAF函数UDF函数scala> val df=spark.read.json("people.json")df: org.apache.spark.sql.DataFrame = [age: bigint, name: string]scala> df.show+---+------+|age| name|+---+------+| 30| ...

数据ip.txtaccess.logimport java.io.{BufferedReader, FileInputStream, InputStreamReader}import java.sql.{Connection, DriverManager, PreparedStatement}import org.apache.spark.{SparkConf, SparkCont...