【spark】pyspark kerberos 案例，即pyspark-utils客户端工具类

原创

于 2024-11-25 11:19:51 发布 · 369 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #大数据 #python

xml_utils.py文件：

# -*- coding: utf-8 -*-
import xml.etree.ElementTree as ET

def parse_xml(file_path):
    map = {
   
   }
    tree = ET.parse(file_path)
    root = tree.getroot()
    props = root.findall("property")
    for prop in props:
        name = prop.find('name').text
        value = prop.find('value').text
        map[name] = value
    return map

# test only
if __name__ == '__main__':
    xml_file = 'D:\hive\conf\hive-site.xml'
    map = parse_xml(xml_file)
    print(map)

spark_utils.py文件：

# -*- coding: utf-8 -*-
import os
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
from src.xml_utils import parse_xml

metastore_uris_key = "hive.metastore.uris"
metastore_princiapl_key = "hive.metastore.kerberos.principal"
metastore_sasl_enabled_key = "hive.metastore.sasl.enabled"