知识的力量

1、代码实现import org.apache.spark.sql.SQLContextimport org.apache.spark.{SparkConf, SparkContext}//case class一定要放到外面case class Person(id: Int, name: String, age: Int)object InferringSchema { def

"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7534 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Fi

package com.sdcetimport org.apache.spark.rdd.RDDimport org.apache.spark.{SparkConf, SparkContext}/** * Created by Administrator on 2017/2/10. */object PageRank { def main(args: Array[Strin

package com.galaxy.fym.algorithm.maxsublist;import java.math.BigDecimal;import java.util.*;/** * Created by fengyiming on 2017/2/17. * * @author fengyiming * 随机产生红包：金额正太分布 * *

"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7532 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Fi

package sqlTextimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.sql.SQLContextimport org.apache.spark.sql.DataFrameimport org.apache.spark.sql.types.{StringType, StructFie

package sqlTextimport org.apache.spark.rdd.RDDimport org.apache.spark.sql.SQLContextimport org.apache.spark.{SparkConf, SparkContext}import org.apache.spark.sql.types.{StructType, IntegerType, St

Spark 大数据处理技术整体架构图

Spark快速大数据分析

package sparkExampleimport org.apache.log4j.{Level, Logger}import org.apache.spark.rdd.RDDimport org.apache.spark.{SparkConf, SparkContext}/** * Created by xiaoxu */object SparkRDDOperation

原文来自于：http://blog.youkuaiyun.com/dabokele/article/details/52810573本文基于Spark Streaming Programming Guide原文翻译， 加上一些自己的理解和小实验的结果。 　　一、概述　　Spark Streaming是基于Core Spark API的可扩展，高吞吐量，并具有容错能力的用

原文：http://blog.youkuaiyun.com/dabokele/article/details/52602412根据Spark官方文档中的描述，在Spark Streaming应用中，一个DStream对象可以调用多种操作，主要分为以下几类TransformationsWindow OperationsJoin OperationsOutput Operations

"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7539 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Fi

[root@hadoop1 spark]# spark-sql --master spark://hadoop1:7077,hadoop2:7077 --executor-memory 1g --total-executor-cores 2 --driver-class-path /usr/local/hive/lib/mysql-connector-java-5.1.35-bin.jar l

"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7534 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Fi

import java.sql.DriverManagerimport org.apache.spark.rdd.JdbcRDDimport org.apache.spark.{SparkConf, SparkContext}object JdbcRDDDemo { def main(args: Array[String]) { System.setProperty("ha

"F:\Program Files\Java\jdk1.7.0_15\bin\java" -Didea.launcher.port=7533 "-Didea.launcher.bin.path=F:\Program Files (x86)\JetBrains\IntelliJIDEASpark\bin" -Dfile.encoding=UTF-8 -classpath "F:\Program Fi

package com.sdcetimport org.apache.spark.{SparkConf, SparkContext}/** * Created by Administrator on 2017/2/16. */object TestColl { System.setProperty("hadoop.home.dir", "E:\\winutils-hadoop-

推荐引擎系统结构

package com.sdcetimport org.apache.spark.sql.SQLContextimport org.apache.spark.{SparkConf, SparkContext}/** * Created by Administrator on 2017/2/18. */object JdbcText { def main(args: Arra

原文来自于：http://blog.youkuaiyun.com/dabokele/article/details/52802150　Spark SQL中的DataFrame类似于一张关系型数据表。在关系型数据库中对单表或进行的查询操作，在DataFrame中都可以通过调用其API接口来实现。可以参考，Scala提供的DataFrame API。　　本文中的代码基于Spark-

Spark_Bench使用文档1. 什么是Spark-BenchSparkBench是Spark的基准测试组件(集成了很多spark支持的经典测试案列)。 它大致包含四种不同类型的测试案例，包括机器学习，图形处理，流处理和SQL查询。Spark-Bench所选择的测试案例可以，在不同的工作负载情况下测试出系统瓶颈; 目前，我们大致涵盖了CPU，内存和Shuffle以及IO密