心如止水 宠辱不惊

转：http://blog.youkuaiyun.com/program_anywhere/article/details/53728186学习路上的各种坑： 1、在new对应的栏目中没有Scala class选项 2、解决 这是因为maven的配置问题 file->settings->Build,Excution->Build Tools->Maven now do： 去掉选

import java.net.URLimport org.apache.spark.{SparkConf, SparkContext}/** * 根据指定的学科, 取出点击量前三的 * Created by root on 2016/5/16. */object AdvUrlCount { def main(args: Array[String]) { //

在Linux中有一个级强大的网络工具netcat，在默认情况下面都是没有安装的，现在介绍一下安装过程 其实安装很简单切换到root用户：yum install -y nc测试使用 在一个终端上 输入 nc -lk 9999 输入数据。。。新开启一个终端 输入 nc ip:9999 就可以看到上一个终端输入的数据

org.apache.spark spark-streaming_2.10 ${spark.version} org.apache.spark spark-streaming-kafka_2.10 1.6.1

import java.net.URLimport org.apache.spark.{HashPartitioner, Partitioner, SparkConf, SparkContext}import scala.collection.mutable/** * Created by root on 2016/5/18. */object UrlCountPartiti

import org.apache.spark.streaming.{Seconds, StreamingContext}import org.apache.spark.{HashPartitioner, SparkConf, SparkContext}/** * Created by root on 2016/5/21. */object StateFulWordCount {

import org.apache.spark.SparkConfimport org.apache.spark.streaming.flume.FlumeUtilsimport org.apache.spark.streaming.{Seconds, StreamingContext}/** * Created by ZX on 2015/6/22. */object Flum

import java.net.InetSocketAddressimport org.apache.spark.SparkConfimport org.apache.spark.storage.StorageLevelimport org.apache.spark.streaming.flume.FlumeUtilsimport org.apache.spark.streaming.{

import org.apache.spark.{SparkConf, SparkContext}object OrderContext { implicit val girlOrdering = new Ordering[Girl] { override def compare(x: Girl, y: Girl): Int = { if(x.faceValue >

import java.io.{BufferedReader, FileInputStream, InputStreamReader}import scala.collection.mutable.ArrayBufferobject IPLocationDemo { def ip2Long(ip: String): Long = { val fragments = ip.sp

import org.apache.spark.{SparkConf, SparkContext}/** * Created by root on 2016/5/18. */object IPLocation { def ip2Long(ip: String): Long = { val fragments = ip.split("[.]") var ipNum

package cn.itcast.spark.day5import org.apache.spark.storage.StorageLevelimport org.apache.spark.{HashPartitioner, SparkConf}import org.apache.spark.streaming.kafka.KafkaUtilsimport org.apache.spa

package cn.itcast.spark.day3import java.sql.DriverManagerimport org.apache.spark.rdd.JdbcRDDimport org.apache.spark.{SparkConf, SparkContext}/** * Created by ZX on 2016/4/12. */object Jdbc

package cn.itcast.spark.day5import org.apache.spark.SparkConfimport org.apache.spark.streaming.{Milliseconds, Seconds, StreamingContext}/** * Created by ZX on 2016/4/19. */object WindowOpts

import java.util.concurrent.ExecutorService;import java.util.concurrent.Executors;public class ThreadPool { public static void main(String[] args) throws Exception { ExecutorService pool = Exec

17/08/09 15:13:32 INFO spark.SparkContext: Created broadcast 0 from textFile at WordCount.scala:15Exception in thread "main" java.io.IOException: Failed on local exception: com.google.protobuf.Inval

转：http://blog.youkuaiyun.com/guotong1988/article/details/50555185reduce(binary_function) reduce将RDD中元素前两个传给输入函数，产生一个新的return值，新产生的return值与RDD中下一个元素（第三个元素）组成两个元素，再被传给输入函数，直到最后只有一个值为止。val c = sc.paral

start-all.sh -> start-master.sh  -> start-slaves.sh//Master启动的脚本start-master.sh  -> spark-daemon.sh start org.apache.spark.deploy.master.Master//Worer的启动过程salves.sh   ->  通过读取slaves 通过ssh的

转：http://blog.youkuaiyun.com/wind520/article/details/46822969

转：http://www.cnblogs.com/zlslch/p/5723764.html  参考

Spark调试：本地模式调试集群模式调试第一种：val conf = new SparkConf().setAppName("WC").setMaster("local[2]")第二种import org.apache.spark.{SparkConf, SparkContext}/** * Created by root on 2016/5/1

转：http://blog.youkuaiyun.com/lsshlsw/article/details/48627737与map方法类似，map是对rdd中的每一个元素进行操作，而mapPartitions(foreachPartition)则是对rdd中的每个分区的迭代器进行操作。如果在map过程中需要频繁创建额外的对象(例如将rdd中的数据通过jdbc写入数据库,map需要为每个元素创建一个

转：http://blog.youkuaiyun.com/huanbia/article/details/51318278问题描述当使用SecureCRT来打开Spark-shell的时候，有时会出现如下问题，当输错的时候想要按Backspace（退格键）或“Delete(删除键)”的时候，无法删除前面删除的内容。 解决方法问题主要出现在我们的SecureC

参考：http://www.tuicool.com/articles/bQVRryr/**   * Mark this RDD for checkpointing. It will be saved to a file inside the checkpoint   * directory set with `SparkContext#setCheckpointDir` and all

在Spark SQL中SQLContext是创建DataFrames和执行SQL的入口，在spark-1.5.2中已经内置了一个sqlContext1.在本地创建一个文件，有三列，分别是id、name、age，用空格分隔，然后上传到hdfs上hdfs dfs -put person.txt / 2.在spark shell执行下面命令，读取数据，将每一行的数据使用列分隔符分割

首先在maven项目的pom.xml中添加Spark SQL的依赖。dependency>    groupId>org.apache.sparkgroupId>    artifactId>spark-sql_2.10artifactId>    version>1.5.2version>dependency>package cn.itcast.spark.sql

package cn.itcast.spark.sqlimport org.apache.spark.sql.{Row, SQLContext}import org.apache.spark.sql.types._import org.apache.spark.{SparkContext, SparkConf}/** * Created by ZX on 2015/12/11.

//1.读取数据，将每一行的数据使用列分隔符分割val lineRDD = sc.textFile("hdfs://node1.itcast.cn:9000/person.txt", 1).map(_.split(" "))//2.定义case class（相当于表的schema）case class Person(id:Int, name:String, age:Int)

转：zx老师########################################alter database hive character set latin1;ALTER TABLE hive.* DEFAULT CHARACTER SET latin1;########################################1.安装hiveC

import org.apache.spark.{SparkConf, SparkContext}/** * Created by root on 2016/5/16. */object ForeachDemo { def main(args: Array[String]) { val conf = new SparkConf().setAppName("Foreach

http://homepage.cs.latrobe.edu.au/zhe/ZhenHeSparkRDDAPIExamples.htmlmap是对每个元素操作, mapPartitions是对其中的每个partition操作val rdd = sc.parallelize(List(1,2,3,4,5),2)----------------------------------------

import java.net.URLimport org.apache.spark.{SparkConf, SparkContext}/** * 取出学科点击前三的 * Created by root on 2016/5/16. */object UrlCount { def main(args: Array[String]) { val conf = new

CentOS6.7-64bit编译hadoop2.6.41.下载maven（apache-maven-3.3.3-bin.tar.gz）http://archive.apache.org/dist/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz2.安装maventar -zxvf apache-maven-3

http://www.cnblogs.com/shoufengwei/p/5619419.htmlhttps://docs.geotrellis.io/en/latest/tutorials/quickstart.html