hive-the summaries of features

  in these days ,i learned to the data warehouse framework-hive ,mainly from the ebook 'programming hive' [1],as it's about 23 chapters in detail;)

  so below are the outlines about this topic:

 

1.overview

2.architecture

3.features

4. hive vs pig,hive vs hbase

5.use cases

 

1.overview

  a data warehouse ,as you think ,one of the most explict features is data scale,may be gb,tb and event pb is a common case.in general ,we want to load the data into a database then query it by SQL ,finally generate a report to give out a decision etc.

  yep,similar to common rmdbs,hive supplies a SQL-style query language(HSQL which will explain the sqls to issue mapreduce jobs),in addition ,hive can use misc fs like local-fs,dfs,hbase cassandra as its underlying essential storage system . 

  so u maybe guess that hive is only a terminal that lies on a dfs.so it is very easy to load or process huge numbe of data via hive tool.

 

2.architecture

  

   figure 1 from 'programming hive'[1]

 

    figure from 'hive-a warehousing framework from facebook'[2]

 

  integrate both fitures above,we know that hive supplies some useful operation interfaces: cli,jdbc/odbc,web gui.and the  thrift server is used to construct a bridge between jdbc/odbc and hive core modules.

  secondly,there is a named 'metastore' machism which used to store table schema etc meta data,ie. derby db by default.as in generaly these info is small enough to place there .but u should use a spare db solutions like mysql cluster if you want to aoid a SPOF (single point  of failure) .

 

3.features

 here are certain important features referenced from 'programming hive':

  sql-style execution language

  the main sql grammers are introduced to hive,so query from  and load to hive are all no learning curves as who has basic sql knowledges;and it's easy to concise sqls command than mapreduce code to issue jobs.

  flexable,controllabe 

  in hive,there are certain execution modes to execute the sql commands.in local mode,it's fit for analyzing small data processes,as this will spawn local job to do that other than real cluster running;in parallel mode,the jobs for the same goal(bussiness) are allowed to run parallelly if that some tasks are independent;in restrict mode,some cost-heavy operations are disallowed to spawn,so this will void to preempt large resources which other jobs are necessary.

  static data 

  unlike a command rmdbs,hive doesn't support row-level DDL to operate on data,once the data are placed in,no updates are allowed;instead,u can use some DML to mantian it some schema related cases.

  own table stucture/store but not rapid query reponse time and not supports transactions

  yep,hive can be integrated with other no-sql solutions,like hbase,cassandra etc,but if u are not happy with them ,u can use the hive-own table to store data query from underlying fil system.

  but unlike nosql,hive will not support effect query responses,as it 's desinged to analyze large scale data rather than a common db,this is why hive comes from.

  partitioned table

  with this ,the data generated from hive can be scaled horizontally across the cluster.

  supports views and indexes

  hive will use a extended table to support indexes,like hbase,this is a usual solution .

  file compression

  similar to hadoop 's compress ,the output of hive is also able to use compress algorithms to reduce io floods.

 

  

4. hive vs pig ,hive vs hbase

  hive vs pig

	hive	pig
execution language	sql	pig latin
own db	yes(so it supports jdbc/odbc connectors)	no
execution model	compile(to MR),optimize,execute	same as hive

 

  hive vs hbase

 

	hive	hbase
support sql	yes	no
transaction	no	row-level(not confirm above 1.x for table -level)
real response	no	yes
target	data warehouse	no-sql
index	yes	no
file system	hdfs,mapR..	hdfs

 

 

5.use cases

  note that above 2.x the hive jar is complied with jdk7,so if u deploy it to a jdk6 or below ,it will complain the 'unsupport class version for marjor.minor:51.0' error.but u can upgrade to 1.x in the fly.

 

 

 

ref:

[1]programming hive

[2]hive-a warehousing solution from facebook