基于ambari安装Kerberos client

1.安装kerberos依赖包（可以不安装，此为Ubuntu 依赖）
yum install krb5-user libpam-krb5 libpam-ccreds auth-client-config
yum  install  krb5-workstation

2.拷贝kerberos 集群的 /etc/krb5.conf 到本地

 

3.创建 本地client 访问Kerberos集群的代理用户(HTTP),访问hdfs用户(gai)，秘钥 (addprinc  HTTP或gai/livy所在主机名@GAI.COM）
 
在Kerberos  kdc server服务 （Kerberos相关目录带有kdc.conf的机器） 所在机器上，命令输入：

kadmin.local （不在该kdc server机器上，使用  kadmin -p admin/admin@GAI.COM  ，输入admin密码进行后续操作）
执行命令：

addprinc  HTTP/livy.test.com@GAI.COM
addprinc  gai/livy.test.com@GAI.COM

此步根据需要省略：设置keytab过期时间： modprinc -maxrenewlife "1 week" +allow_renewable HTTP/livy.test.com@GAI.COM

此步根据需要省略：设置keytab过期时间： modprinc -maxrenewlife "1 week" +allow_renewable gai/livy.test.com@GAI.COM

xst -k http.livy.test.com.keytab   HTTP/livy.test.com@GAI.COM

xst -k gai.livy.test.com.keytab   gai/livy.test.com@GAI.COM

 

3.拷贝kerberos 集群的 某个keytab 到本地，在程序中指定该目录，用于API访问时权限验证

4.输入密码，初始化 keytab:

kinit dp/admin@GAI.COM

 

5.复制jersey-*-.jar到classpath (否则会报yarn的classNotFound)

 

6.将远程集群的/hadoop/etc/hadoop所有的*.xml配置文件复制到当前项目的classpath(代码编译后产生的目录）下面
使用如下代码查找合适的classpath:
 

 

7.在kdc server 创建对应的 本地（提交livy作业的host）主机的 principal HTTP/_HOSTNAME（必须相互设置，远程集群每台机器有本地的hostname ip ，本地也有远程集群的所有机器hostname-ip

8.添加本地（livy所在机器）访问带有Kerberos的hdfs集群的代理设置（设置在core-site.xml文件中）：（否则会报如下错误）

hadoop.proxyuser.dp.groups=*
hadoop.proxyuser.dp.hosts=10.111.23.70
hadoop.security.authentication=kerberos

 

 

019-03-21 12:18:20 ERROR ApplicationMaster:91 - Uncaught exception: 
org.apache.spark.SparkException: Failed to connect to driver!
    at org.apache.spark.deploy.yarn.ApplicationMaster.waitForSparkDriver(ApplicationMaster.scala:657)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:517)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:800)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
    at org.apache.spark.deploy.yarn.ApplicationMaster.doAsUser(ApplicationMaster.scala:799)
    at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:259)
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:824)
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:854)
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala)
2019-03-21 12:18:20 INFO  ApplicationMaster:54 - Final app status: FAILED, exitCode: 13, (reason: Uncaught exception: org.apache.spark.SparkException: Failed to connect to driver!)
2019-03-21 12:18:20 INFO  ApplicationMaster:54 - Deleting staging directory hdfs://namenode.gai.test.com:8020/user/dp/.sparkStaging/application_1553139449000_0004
2019-03-21 12:18:20 INFO  ShutdownHookManager:54 - Shutdown hook called