Talk Is Cheap

否则报错 WHERE clause must be a condition。注意where那里要加。

INode毫无疑问断定就是web页面那个50070端口看到的目录BlocksMap类: 保存数据块到INode和DataNode的映射关系 BlocksMap存放于fsimage文件中。fsimage所在位置：对于hadoop1.x是由hdfs-site.xml中hadoop.tmp.dir来配置的BP是Block Pool的缩写Block的实际

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSCommands.htmlhttp://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/FileSystemShell.html

core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value></property><property> <name>io.file.buffer.size</name> <v

val thePath = new Path(theHdfsPath) val hdfs = org.apache.hadoop.fs.FileSystem.get( new java.net.URI(THE_MASTER_URI), new org.apache.hadoop.conf.Configuration()) val writer = new PrintWriter(hdfs.create(thePath)) writer.println...

https://issues.apache.org/jira/browse/HDFS-2576

替换bin/hdfs里的相应的启动配置 如果不替换直接添加，会有ERROR: Cannot load this JVM TI agent twice, check your java command line for duplicate jdwp options. Error occurred during initialization of VM agent library failed t

HDFS是一个分布式文件系统，在普通的价格低廉的商用机器上运行，具有很高的容错性，读写性能很高，比较适用于大块文件，HDFS对外提供POSIX标准的流接口来读写文件。 1.设计目标 1.1宕机容忍 一个HDFS集群会包含数百上千个机器，每个都存储整个HDFS的一部分数据，如果有宕机情况，快速的侦测宕机和自动恢复是HDFS架构的主要设计目标。 1.2流式数据访问 HDFS设计目标更多是为批量

走 hadoop-common-project\hadoop-common\src\main\java\org\apache\hadoop\ipc\Server.java 的 value = call(call.rpcKind, call.connection.protocolName, call.rpcRequest, call.time

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/core-default.xmlhttp://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Tachyon的设计思想主要有两个，第一个是基于内存的OffHeap的分布式存储，就是一定要把数据存储在JVM堆空间之外，这样的话能够避免GC。第二个是通过在存储层保存数据的Lineage实现容错，这是在Spark中引入的思想，Lineage记录了源数据以及源数据经过什么样的计算得到的当前数据，Tachyon将这些本来是在计算层才关心的信息放到了存储层。Tachyon仅仅保存一份数据在内存中，内存

mvn -Dhadoop.version=2.3.0 clean install一般是maven下载失败，断了再敲一遍继续下载相应的库

一定要自己编译 mvn clean package -DskipTests -Dhadoop.version=2.5.0 这个version是>hadoop version里看到的hdfs里加上 http://tachyon-project.org/Running-Hadoop-MapReduce-on-Tachyon.html 的配置tachyon/conf里面配置端口是hdfs节点的浏览

读类型：CACHE – 读取数据并缓存在本地内存 NO_CACHE – 读取数据但不缓存在本地内存写类型：MUST_CACHE – 只写本地内存，空间不足时报ERROR TRY_CACHE – 只写本地内存，空间不足时报WARNING THROUGH – 只写UFS CACHE_THROUGH – 同时写本地内存和UFS（TRY_CACHE + THROUGH） ASYNC_THROUGH

package test;import java.io.IOException;import tachyon.TachyonURI;import tachyon.client.FileOutStream;import tachyon.client.InStream;import tachyon.client.ReadType;import tachyon.client.TachyonFS;

public void map(Object key, Text value, Context context ) throws IOException,InterruptedException {//每个mapper对于每个key-value，一次调用map读进一个key-value StringTokenizer itr = new Strin

第三方jar放在$HADOOP_HOME/lib下，需重启在$HADOOP_HOME/bin-mapreduce1下./hadoop jar cloudwave-test.jar com.cloudwave.mapreduce.CloudwaveMapReduceTest

看MapTask类的runNewMapper方法

MapReduce程序可以以以下三种模式运行Local(Standalone) Mode：只有一个 Java 虚拟机在跑，完全没有分布式的成分。且不使用HDFS文件系统，而是使用本机的Linux文件系统。Pseudo-distributed Mode：在同一台机器上启动独立数个 JVM 进程，每一个hadoop daemon运行在一个单独的JVM进程中，进行“伪分布式”操作。

http://www.powerxing.com/hadoop-build-project-by-shell/

http://hstore.cs.brown.edu/documentation/deployment/mapreduce/#MapInputQuery

select * from table_name lateral view explode(array_int_column) a as int_column where int_column = 123

如果两个string column做笛卡尔积的jaccard距离计算，即使按 https://blog.youkuaiyun.com/guotong1988/article/details/111594617计算，速度也是很慢，所以要先做一步粗召回，用string column1的前几个字（或者ngram）作为key1，用string column2的前几个字（或者ngram）作为key2，然后key1 join key2，即得到粗召回的结果，再每行计算jaccard距离...

只能先collect小表数据再broadcast，然后再每个map里match之后break

import org.apache.spark.sql.expressions.Windowimport org.apache.spark.sql.functions.{rand, row_number}val windowFun = Window.partitionBy("groupby_column").orderBy(col("score").desc)val resultDF = dataDF.withColumn("rank", row_number.over(windowFun))

https://stackoverflow.com/questions/32435263/dataframe-join-optimization-broadcast-hash-joinimport org.apache.spark.sql.functions.broadcastlargeDataframe.join(broadcast(smallDataframe), ...)

从 https://github.com/agile-lab-dev/sparksearchengine 下载源码，复制进自己的项目import it.agilelab.bigdata.spark.search.SearchableRDDimport it.agilelab.bigdata.spark.search.dsl._import it.agilelab.bigdata.spark.search.impl.analyzers.{DefaultAnalyzer, EnglishWikipedi

写出小testcaseobject Test { def main(args: Array[String]): Unit = { val tmpArrayList = new util.ArrayList[JSONObject]() val tmpMap = new JSONObject(true) tmpMap.put("id",1) tmpMap.put("name","hello") tmpArrayList.add(tmpMap) print(J

join之前重命名column，join还是bug，根本不行。

用df.write.mode(SaveMode.Overwrite).saveAsTable(“XXX”)

下载spark-2.2.0-bin-hadoop2.6.tgz然后进bin里运行./spark-shell然后

val actualDF = sourceDF.withColumn( "word1_word2_levenshtein", levenshtein(col("word1"), col("word2")))actualDF.show()+------+-------+-----------------------+| word1| word2|word1_word2_leve...

val rdd1 = sc.parallelize(Seq(("one two three four five six seven"), ("one two three four five six seven"), ("one two three four five six seven")))然后rdd1.map(_.split(" ")).collect结果Array[Arra...

因为join的两个key重名了，结果是右表的key列。

def intersectFn = udf { (document0: String, document1: String) => val set1 = document0.split("@@@").toSet val set2 = document1.split("@@@").toSet val intersect_set = set1.interse...

import org.apache.spark.sql.functionsval jdf = df1.join(df2,functions.levenshtein(df2("str_col1"),df1("str_col2"))<5)

前面有个join，可能是join的两边重复的key太多了。

wordcount例子val conf = new SparkConf().setAppName("GroupAndReduce").setMaster("local")val sc = new SparkContext(conf)val words = Array("one", "two", "two", "three", "three", "three")val wordsRDD = ...

http://svn.apache.org/repos/asf/hadoop/

Configuration config = new Configuration();  FileSystem fs = FileSystem.get(config);  DistributedFileSystem hdfs = (DistributedFileSystem) fs;  DatanodeInfo[] dataNodeStats = hdfs.getDataNodeSta