Running Spark Streaming Jobs on a Kerberos-Enabled Cluster

最新推荐文章于 2021-04-21 18:03:06 发布

转载最新推荐文章于 2021-04-21 18:03:06 发布 · 70 阅读

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/felixzh/p/10558559.html

本文介绍在启用Kerberos的集群上运行Spark Streaming作业的步骤，包括选择或创建用户账户、生成keytab、创建JAAS登录配置文件、添加配置设置、通过YARN本地资源分发文件，还给出了相关配置示例及在spark - submit命令中传递文件和配置选项的方法。

Use the following steps to run a Spark Streaming job on a Kerberos-enabled cluster.

Select or create a user account to be used as principal.
This should not be the kafka or spark service account.
Generate a keytab for the user.
Create a Java Authentication and Authorization Service (JAAS) login configuration file: for example, key.conf.
Add configuration settings that specify the user keytab.
The keytab and configuration files are distributed using YARN local resources. Because they reside in the current directory of the Spark YARN container, you should specify the location as ./v.keytab.

The following example specifies keytab location ./v.keytab for principal vagrant@example.com:
```
KafkaClient {
   com.sun.security.auth.module.Krb5LoginModule required
   useKeyTab=true
   keyTab="./v.keytab"
   storeKey=true
   useTicketCache=false
   serviceName="kafka"
   principal="vagrant@EXAMPLE.COM";
};
```

In your spark-submit command, pass the JAAS configuration file and keytab as local resource files, using the --filesoption, and specify the JAAS configuration file options to the JVM options specified for the driver and executor:

spark-submit \
    --files key.conf#key.conf,v.keytab#v.keytab \
    --driver-java-options "-Djava.security.auth.login.config=./key.conf" \
    --conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./key.conf" \
...

Pass any relevant Kafka security options to your streaming application.
For example, the KafkaWordCount example accepts PLAINTEXTSASL as the last option in the command line:
```
KafkaWordCount /vagrant/spark-examples.jar c6402:2181 abc ts 1 PLAINTEXTSASL
```