术业有专攻

  Since MapReduce and HDFS are complex distributed systems that run arbitrary user code, there’s no hard and fast set of rules to achieve optimal performance; instead, I tend to think of tuning a cl...

                    References https://github.com/pranab/sifarish

hadoop 2..30的官方tarball中 ./lib/native中的库只适合32位操作系统，在64位下安装会报一些错误，使用hadoop启动不起来。所以需要在64位上重新编译。 1. enviroment    hadoop 2.3.0    ubuntu 12.04 64   2. follow these steps to recompile hadoop sudo a...

1. when put local file to HDFS using #hadoop fs -put in.txt /test, there is a error message: hdfs.DFSClient: Exception in createBlockOutputStream java.net.NoReouteToHostException solution...

When I submit a java job (include some  map/reduce jobs) in the hue UI using oozie Editor,  the third party jars are not loaded correctly.   1. the only success way i used is to build a fat jar wh...

http://stackoverflow.com/questions/12956488/hadoop-nosuchmethodexception

import java.io.IOException; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; im...

 ENV: Ubuntu 12.04  1. Install Eclipse 2. create desktop shortcut for Eclipse     a. create an empty document named eclipse.xx     b. edit eclipse.xx like followings (avoid failing to open eclip...

                    Reference https://blogs.oracle.com/datawarehousing/entry/integrating_hadoop_data_with_o

In my work, I run a situation that I want to use A mapper reading a file with to fields (questionId, questionTags) and outpute format likes key: questionId  value: questionTags, while B mapper readi...

http://hadoopecosystemtable.github.io/ http://blog.andreamostosi.name/big-data/ https://github.com/youngwookim/awesome-hadoop

MySQL Applier for Hadoop Replication via the Hadoop Applier is implemented by connecting to the MySQL master and reading binary log events as soon as they are committed, and writing them into a fil...

http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-1.html MySQL replication enables data to be replicated from one MySQL database server (the master) to one or more MySQL da...

http://innovating-technology.blogspot.com/2013/04/mysql-hadoop-applier-part-2.html This is a follow up post, describing the implementation details of Hadoop Applier, and steps to configure and insta...

1.install and configure hadoop-2.6.0 ($HADOOP_HOME must be set). 2. download mysql-5.6.22.tar.gz source code from http://dev.mysql.com/downloads/mysql/  #tar xf mysql-5.6.22.tar.gz #cd mysql-5.6....

HDFS is append only, yes. The short answer to your question is that, to modify any portion of a file that is already written, one must rewrite the entire file and replace the old file."Even for a sin...

We are using Sqoop to export data from the hive to SQL Server. The new data is always appended to the existing data in SQL Server. Is it possible to truncate the SQL Server table via Sqoop before...

http://techkites.blogspot.com/2014/06/real-time-clickstream-analytics-using.html

                  Some example codes here https://github.com/adamjshook/mapreducepatterns/tree/master/MRDP/src/main/java/mrdp   https://github.com/adamjshook/mapreducepatterns/blob/maste...

The number of map tasks for a given job is driven by the number of input splits and not by the mapred.map.tasks parameter. For each input split a map task is spawned. So, over the lifetime of a mapr...

0 reducer means reduce step will be skipped and mapper output will be the final out Identity reducer means then shuffling/sorting will still take place If you do not need sorting of map results -...

                    References http://stackoverflow.com/questions/2499585/chaining-multiple-mapreduce-jobs-in-hadoop https://developer.yahoo.com/hadoop/tutorial/module4.html#chaining ht...

1.when i create a hive table in hue, there errors comes   Solution:#hadoop dfsadmin -safemode leave http://www.linkedin.com/groups/Creating-table-in-Hive-getting-4547204.S.225243871   2.error ...

  Redis is a very cool open-source key-value store that can add instant value to your Hadoop installation. Since keys can contain strings, hashes, lists, sets and sorted sets, Redis can be used a...

hadoop-env.sh Must set JAVA_HOME in namenode and secondary namenodes, or the start-dfs.sh will run errors

Installing hadoop 2.2.0 clusters with 3 nodes(one for namenode/resourcemanager and secondary namenode while the other tow nodes for datanode/nodemanager) 1. ip assignments     192.168.122.1       ...

                  References http://blog.youkuaiyun.com/lastsweetop/article/details/9001467  

Reduce-side joining / repartitioned sort-merge join     Note:DataJoinReducerBase, on the other hand, is the workhorse of the datajoin package, and it simplifies our programming by performing a fu...

  http://www.cnblogs.com/zhangchaoyang/articles/2647905.html http://blog.pureisle.net/archives/1618.html http://www.youkuaiyun.com/article/2014-01-01/2817984-13-tools-let-hadoop-fly http://blog.mortar...

https://gridgaintech.wordpress.com/2013/11/07/hadoop-100x-faster-how-we-did-it/   Almost two years ago, Dmitriy and I stood in front of a white board at GridGain’s office thinking: “How can we deli...

tungsten-replicator-3.0.0-524-src

    http://blog.youkuaiyun.com/menergy/article/details/17583823

  when i use there is a error java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/MorphlineCompilationException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Cla...

https://bitbucket.org/winebarrel/mysql-replication-listener  http://flume.apache.org/FlumeUserGuide.html#a-simple-example https://www.cyberagent.co.jp/recruit/techreport/report/id=7474 https://do...