sqoop2 :Install and Basic Usage

Sqoop2 Install

1. install server

 download the tarball form the official website

 #tar -xzvf sqoop-1.99.3-bin-hadoop200.tar.gz

 Assume that the server and client will install in the same host:192.168.122.1

 configure  server  related configuration files in dir

/path/to/sqoop-1.99.3-bin-hadoop200/server/conf

 a. catalina.properties

 add hadoop related jars to common.load property

 b. sqoop.properties

 the most import item is

org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/path/to/hadoop-conf-dir

  To avoid access privileges, we should copy all hadoop configuration files to another dir. The reason is that the hadoop namode and yarn resource manager damons run in the same host with sqoop server, these conf files will be locked by hadoop, which lead to access denied to sqoop server.

 

2. install client

I installed the sqoop server and client in the same host due to resource limilation, which I used one PC installed ubuntu 12.04 with 3 centos VM.

 

3. add env

add sqoop bin dir to path in ~/.bashrc  and using source to make it effect immidately.

4. start server

 # sqoop.sh server start

5. start client

#sqoop.sh client

or you can run sqoop script in batch mode

#sqoop.sh client /path/to/script.sqoop

Note:Commands create, update and clone are not currently supported in batch mode

so, you can't create/update/clone some connection and job in you scripts. Also, you can't

use sqoop1's commands in script,such as import.

 

sqoop:000> set server --host your.host.com --port 12000 --webapp sqoop

sqoop:000> show version --all

sqoop:000> show connector --all

sqoop:000> create connection --cid 1

sqoop:000> show connector --all

sqoop:000> create job --xid 1 --type import

sqoop:000> start job -j 1

sqoop:000> status job -j 1

sqoop:000> update job -j 1

sqoop:000> clone job -j 1

sqoop:000> delete job -j 1

the usage of sqoop CLI refer to the official documents.

 

Warning:Sqoop2 is totally different from Sqoop1. So the commands belong to sqoop1 could not be used

in Sqoop2 CLI or script.

 

Sqoop2  wiki:

https://cwiki.apache.org/confluence/display/SQOOP/Home

https://cwiki.apache.org/confluence/display/SQOOP/Sqoop2+Quickstart

 

Sqoop1 Usage: https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html