python-sparksql 报错java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST

本文链接：https://blog.youkuaiyun.com/PingChangYu/article/details/84666782

在尝试使用Python和pyspark执行SparkSQL时遇到java.util.NoSuchElementException，错误源于缺少环境变量_PYSPARK_DRIVER_CALLBACK_HOST和_PYSPARK_DRIVER_CALLBACK_PORT。分析源码发现在某些版本的pyspark中未定义这两个变量。解决方案是安装特定版本的pyspark，例如安装版本XX，之后成功运行并确认版本。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

环境

Python version 2.7.5
spark version 2.2.1

报错

打算使用python调用pyspark包执行sparksql

首先使用pip install pyspark安装pyspark包，然后执行以下python脚本报错

from pyspark import SparkConf,SparkContext
from pyspark.sql import HiveContext
conf = SparkConf().setMaster("local").setAppName("test")
sc = SparkContext(conf=conf)`
hc = HiveContext(sc)

报错信息:

Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/11/30 18:10:46 ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.util.NoSuchElementException: key not found: _PYSPARK_DRIVER_CALLBACK_HOST
        at scala.collection.MapLike$class.default(MapLike.scala:228)
        at scala.collection.AbstractMap.default(Map.scala:59)
        at scala.collection.MapLike$class.apply(MapLike.scala:141)
        at scala.collection.AbstractMap.apply(Map.scala:59)
        at org.apache.spark.api.python.PythonGatewayServer$$anonfun$main$1.apply$mcV$sp(PythonGatewayServer.scala:50)
        at org.apache.spark.util.Utils$.tryOrExit(Utils.scala:1262)
        at org.apache.spark.api.python.PythonGatewayServer$.main(PythonGatewayServer.scala:37)
        at org.apache.spark.api.python.PythonGatewayServer.main(PythonGatewayServer.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:777)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/pyspark/context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "/usr/lib/python2.7/site-packages/pyspark/context.py", line 298, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gate