<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[duzewen的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/duzewen</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; duzewen]]></copyright><item><title><![CDATA[创建分区表t]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/74853421</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/74853421</guid><author>duzewen</author><pubDate>Sat, 08 Jul 2017 22:13:58 +0800</pubDate><description><![CDATA[CREATE TABLE t(id int,name string,age int) PARTITIONED BY (Year INT, Month INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' ;]]></description><category></category></item><item><title><![CDATA[hive基本操作]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/74853371</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/74853371</guid><author>duzewen</author><pubDate>Sat, 08 Jul 2017 22:05:44 +0800</pubDate><description><![CDATA[hive命令行操作
------------------------
 1.创建hive的数据库
  $hive>hive --version    //
  $hive>hive --help     //
  $hive>create database mydb2 ;  //
  $hive>show databases ;
  $hive>use mydb2 ;
  $hiv]]></description><category></category></item><item><title><![CDATA[hive-site.xml]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/74805964</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/74805964</guid><author>duzewen</author><pubDate>Sat, 08 Jul 2017 14:14:34 +0800</pubDate><description><![CDATA[Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyrigh]]></description><category></category></item><item><title><![CDATA[配置hbase本地模式，伪分布模式和完全分布式]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/67096796</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/67096796</guid><author>duzewen</author><pubDate>Mon, 27 Mar 2017 23:41:03 +0800</pubDate><description><![CDATA[1.配jdk环境变量
 
 2.有hadoop集群

 3.tar 开hbase压缩包

 4.配置hbase环境变量

 
 5.验证安装是否成功
  $>hbase version
配置hbase模式
  5.1)本地模式
   [hbase/conf/hbase-env.sh]
   EXPORT JAVA_HOME=/soft/jdk
   [hbase/co]]></description><category></category></item><item><title><![CDATA[配置resourcemanager的HA自动容灾]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/67094885</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/67094885</guid><author>duzewen</author><pubDate>Mon, 27 Mar 2017 23:34:54 +0800</pubDate><description><![CDATA[1.配置yarn-site.xml
  
   yarn.resourcemanager.ha.enabled
   true
  
  
   yarn.resourcemanager.cluster-id
   cluster1
  
  
   yarn.resourcemanager.ha.rm-ids
   rm1,rm2
  
  
   yarn.reso]]></description><category></category></item><item><title><![CDATA[从0开始部署hadoop HDFS的HA集群，使用zk实现自动容灾]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/67092643</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/67092643</guid><author>duzewen</author><pubDate>Mon, 27 Mar 2017 23:33:16 +0800</pubDate><description><![CDATA[我有6台虚拟机s201~s206
其中namenode是s201和s206
 
1.停掉hadoop的所有进程
  
 2.删除所有节点的日志和本地数据.
  xcall.sh "rm -rf /soft/hadoop/logs/*"
                xcall.sh "rm -rf /home/centos/hadoop/*"(查看删干净没有xcall.sh "ls]]></description><category></category></item><item><title><![CDATA[解决hbase时间不同步问题]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/67086771</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/67086771</guid><author>duzewen</author><pubDate>Mon, 27 Mar 2017 23:25:15 +0800</pubDate><description><![CDATA[时间不同步问题:   在每台机子上输入sudo ntpdate time.nist.gov再起hbase集群就行了]]></description><category></category></item><item><title><![CDATA[hive启动问题解决方法]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/65449368</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/65449368</guid><author>duzewen</author><pubDate>Thu, 23 Mar 2017 21:21:56 +0800</pubDate><description><![CDATA[Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby


解决方]]></description><category></category></item><item><title><![CDATA[定义Nvl函数]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919808</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919808</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:17:47 +0800</pubDate><description><![CDATA[import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
 import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
 impo]]></description><category></category></item><item><title><![CDATA[hive自定义函数]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919782</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919782</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:16:58 +0800</pubDate><description><![CDATA[1.创建类，继承UDF
  package com.hivedemo.udf;
  import org.apache.hadoop.hive.ql.exec.Description;
  import org.apache.hadoop.hive.ql.exec.UDF;
  /**
   * 自定义hive函数
   */
  @Description(name = "myadd]]></description><category></category></item><item><title><![CDATA[hive调优]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919758</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919758</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:15:37 +0800</pubDate><description><![CDATA[1.explain
  使用explain查看查询计划
  hive>explain [extended] select count(*) from customers ;
  hive>explain select t.name , count(*) from (select a.name ,b.id,b.orderno from customers a ,orders b where a]]></description><category></category></item><item><title><![CDATA[hive命令]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919642</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919642</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:08:26 +0800</pubDate><description><![CDATA[//创建表,external 外部表
 $hive>CREATE external TABLE IF NOT EXISTS t2(id int,name string,age int) COMMENT 'xx' ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE ;

 //查看表数据
 $hive>desc t]]></description><category></category></item><item><title><![CDATA[mysql的连接查询]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919578</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919578</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:05:02 +0800</pubDate><description><![CDATA[1.准备表[mysql.sql]
 drop table if exists customers; -- 删除表
 drop table if exists orders ; -- 删除表
 create table customers(id int primary key auto_increment , name varchar(20) , age int);     -- 创建cust]]></description><category></category></item><item><title><![CDATA[mysql的隔离级别和出现的问题]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919539</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919539</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 15:03:12 +0800</pubDate><description><![CDATA[隔离级别
----------------
 为了避免出现哪种并发现象的。
 1  //read uncommitted ,读未提交
 2  //read committed ,读已提交
 4  //repeatable read ,可以重复读
 8  //serializable  ,串行化(悲观锁)


演示mysql事务隔离级别
---------------------]]></description><category></category></item><item><title><![CDATA[join操作]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919414</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919414</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:58:08 +0800</pubDate><description><![CDATA[数据
------------
 [customers.txt]
 1,tom,12
 2,tom,13
 3,tom,14
 4,tom,15
 
 [orders.txt]
 1,no001,12.23,1
 2,no001,12.23,1
 3,no001,12.23,2
 4,no001,12.23,2
 5,no001,12.23,2
 6,no001,12.]]></description><category></category></item><item><title><![CDATA[HA高可用]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919346</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919346</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:52:43 +0800</pubDate><description><![CDATA[1.NFS
  网络共享存储设备。
 2.QJM
  Quorum Journal Manager
 3.两个名称节点
  active   //激活
  standby   //待命

active   //激活
deactive  //钝化
SPOF
----------------
 single point of failure,单点故障。
事务是个特性
---]]></description><category></category></item><item><title><![CDATA[自定义机架感知(优化hadoop集群一种方式)]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919306</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919306</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:50:16 +0800</pubDate><description><![CDATA[1.自定义实现类
  package com.it18zhang.hdfs.rackaware;
  import org.apache.hadoop.net.DNSToSwitchMapping;
  import java.util.ArrayList;
  import java.util.List;
  /**
   *机架感知类
   */
  public class ]]></description><category></category></item><item><title><![CDATA[使用DBWritable完成同mysql交互]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919247</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919247</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:47:01 +0800</pubDate><description><![CDATA[1.准备数据库
  create database big4 ;
  use big4 ;
  create table words(id int primary key auto_increment , name varchar(20) , txt varchar(255));
  
  insert into words(name,txt) values('tomas','hello]]></description><category></category></item><item><title><![CDATA[二次排序的实现代码]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919173</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919173</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:43:29 +0800</pubDate><description><![CDATA[Key是可以排序的。
 需要对value排序。
 1.自定义key
 
  import org.apache.hadoop.io.WritableComparable;
  import java.io.DataInput;
  import java.io.DataOutput;
  import java.io.IOException;
  /**
   * 自定义组合ke]]></description><category></category></item><item><title><![CDATA[全排序的实现方法]]></title><link>https://blog.youkuaiyun.com/duzewen/article/details/64919140</link><guid>https://blog.youkuaiyun.com/duzewen/article/details/64919140</guid><author>duzewen</author><pubDate>Wed, 22 Mar 2017 14:40:04 +0800</pubDate><description><![CDATA[有三种方法：


1.定义1个reduce


 2.自定义分区函数.


  自行设置分解区间。


 3.使用hadoop采样机制。

  通过采样器生成分区文件，结合hadoop的TotalOrderPartitioner进行分区划分。
  TotalOrderPartitioner  //全排序分区类,读取外部生成的分区文件确定区间。
  使用时采样代码在最]]></description><category></category></item></channel></rss>