CST spark ERROR - Job run failed! java.lang.Exception: java.lang.RuntimeExceptio

本文记录了一次使用Azkaban执行Spark任务时遇到的问题及解决过程。主要问题是由于缺少必要的配置导致任务启动失败,通过调整配置文件解决了该问题。
11-04-2018 17:22:03 CST spark INFO - Starting job spark at 1523438523394
11-04-2018 17:22:03 CST spark INFO - azkaban.webserver.url property was not set
11-04-2018 17:22:03 CST spark INFO - job JVM args: -Dazkaban.flowid=spark -Dazkaban.execid=110 -Dazkaban.jobid=spark
11-04-2018 17:22:03 CST spark INFO - user.to.proxy property was not set, defaulting to submit user azkaban
11-04-2018 17:22:03 CST spark INFO - Building spark job executor. 
11-04-2018 17:22:03 CST spark INFO - Memory granted for job spark
11-04-2018 17:22:03 CST spark INFO - Not setting up secure proxy info for child process
11-04-2018 17:22:03 CST spark INFO - No classpath specified. Trying to load classes from /root/azkaban/azkaban-exec-server-3.43.0/executions/110
11-04-2018 17:22:03 CST spark INFO - Adding jobtype.classpath: [/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/*:/opt/cloudera/parcels/CDH/lib/spark/conf:/opt/cloudera/parcels/CDH/lib/spark/lib/*]
11-04-2018 17:22:03 CST spark INFO - Using system default spark: /opt/cloudera/parcels/CDH/lib/spark and conf: /opt/cloudera/parcels/CDH/lib/spark/conf
11-04-2018 17:22:03 CST spark INFO - Adding jobtype.global.classpath: [/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/*]
11-04-2018 17:22:03 CST spark INFO - Final classpath: [azkaban.jar, /root/azkaban/azkaban-exec-server-3.43.0/lib/az-core-3.43.0-8-gf0e72fa.jar, /root/azkaban/azkaban-exec-server-3.43.0/lib/azkaban-common-3.43.0-8-gf0e72fa.jar, /root/azkaban/azkaban-exec-server-3.43.0/extlib/azkaban-jobtype-3.0.0.jar, /root/azkaban/azkaban-exec-server-3.43.0/lib/azkaban-hadoop-security-plugin-3.43.0-8-gf0e72fa.jar, /root/azkaban/azkaban-exec-server-3.43.0/executions/110/_resources_spark, /opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/*:/opt/cloudera/parcels/CDH/lib/spark/conf:/opt/cloudera/parcels/CDH/lib/spark/lib/*, /opt/cloudera/parcels/CDH/lib/spark/lib/*, /opt/cloudera/parcels/CDH/lib/spark/conf, /opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/*]
11-04-2018 17:22:03 CST spark INFO - Resolving execution jar/py name: resolvedJarName: /root/azkaban/azkaban-exec-server-3.43.0/executions/110/azkaban.jar
11-04-2018 17:22:03 CST spark INFO - 1 commands to execute.
11-04-2018 17:22:03 CST spark INFO - cwd=/root/azkaban/azkaban-exec-server-3.43.0/executions/110
11-04-2018 17:22:03 CST spark INFO - effective user is: azkaban
11-04-2018 17:22:03 CST spark INFO - Command: java -Dazkaban.flowid=spark -Dazkaban.execid=110 -Dazkaban.jobid=spark -Xms64M -Xmx256M -cp azkaban.jar:/root/azkaban/azkaban-exec-server-3.43.0/lib/az-core-3.43.0-8-gf0e72fa.jar:/root/azkaban/azkaban-exec-server-3.43.0/lib/azkaban-common-3.43.0-8-gf0e72fa.jar:/root/azkaban/azkaban-exec-server-3.43.0/extlib/azkaban-jobtype-3.0.0.jar:/root/azkaban/azkaban-exec-server-3.43.0/lib/azkaban-hadoop-security-plugin-3.43.0-8-gf0e72fa.jar:/root/azkaban/azkaban-exec-server-3.43.0/executions/110/_resources_spark:/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/*:/opt/cloudera/parcels/CDH/lib/spark/conf:/opt/cloudera/parcels/CDH/lib/spark/lib/*:/opt/cloudera/parcels/CDH/lib/spark/lib/*:/opt/cloudera/parcels/CDH/lib/spark/conf:/opt/cloudera/parcels/CDH/lib/hadoop/etc/hadoop:/opt/cloudera/parcels/CDH/jars/* azkaban.jobtype.HadoopSecureSparkWrapper --driver-java-options--masterspark://cdh-master:7077--classcom.azkaban.hollysys.sparkTest--queuedefault/root/azkaban/azkaban-exec-server-3.43.0/executions/110/azkaban.jar
11-04-2018 17:22:03 CST spark INFO - Environment variables: {JOB_OUTPUT_PROP_FILE=/root/azkaban/azkaban-exec-server-3.43.0/executions/110/spark_output_8468954967439774988_tmp, JOB_PROP_FILE=/root/azkaban/azkaban-exec-server-3.43.0/executions/110/spark_props_3032483159170958803_tmp, KRB5CCNAME=/tmp/krb5cc__sparkNoHdfs__spark__spark__110__azkaban, JOB_NAME=spark}
11-04-2018 17:22:03 CST spark INFO - Working directory: /root/azkaban/azkaban-exec-server-3.43.0/executions/110
11-04-2018 17:22:04 CST spark INFO - SLF4J: Class path contains multiple SLF4J bindings.
11-04-2018 17:22:04 CST spark INFO - SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
11-04-2018 17:22:04 CST spark INFO - SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/pig-0.12.0-cdh5.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
11-04-2018 17:22:04 CST spark INFO - SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/slf4j-simple-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
11-04-2018 17:22:04 CST spark INFO - SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/jars/avro-tools-1.7.6-cdh5.11.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
11-04-2018 17:22:04 CST spark INFO - SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.11.0-1.cdh5.11.0.p0.34/lib/spark/lib/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
11-04-2018 17:22:04 CST spark INFO - SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
11-04-2018 17:22:04 CST spark INFO - SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
11-04-2018 17:22:04 CST spark INFO - 18/04/11 17:22:04 INFO root: Args before adjusting driver java opts: [--driver-java-options, , --master, spark://cdh-master:7077, --class, com.azkaban.hollysys.sparkTest, --queue, default, /root/azkaban/azkaban-exec-server-3.43.0/executions/110/azkaban.jar]
11-04-2018 17:22:04 CST spark INFO - Exception in thread "main" java.lang.RuntimeException: Cannot find property [azkaban.link.workflow.url], in Hadoop configuration: [null]
11-04-2018 17:22:04 CST spark INFO - 	at azkaban.jobtype.HadoopJobUtils.javaOptStringFromHadoopConfiguration(HadoopJobUtils.java:586)
11-04-2018 17:22:04 CST spark INFO - 	at azkaban.jobtype.HadoopSecureSparkWrapper.handleDriverJavaOpts(HadoopSecureSparkWrapper.java:168)
11-04-2018 17:22:04 CST spark INFO - 	at azkaban.jobtype.HadoopSecureSparkWrapper.runSpark(HadoopSecureSparkWrapper.java:135)
11-04-2018 17:22:04 CST spark INFO - 	at azkaban.jobtype.HadoopSecureSparkWrapper.main(HadoopSecureSparkWrapper.java:105)
11-04-2018 17:22:04 CST spark INFO - Process completed unsuccessfully in 0 seconds.
11-04-2018 17:22:04 CST spark ERROR - caught error running the job
11-04-2018 17:22:04 CST spark ERROR - Job run failed!
java.lang.Exception: java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException
	at azkaban.jobtype.HadoopSparkJob.run(HadoopSparkJob.java:285)
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:778)
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:594)
	at azkaban.execapp.JobRunner.run(JobRunner.java:555)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:304)
	at azkaban.jobtype.HadoopSparkJob.run(HadoopSparkJob.java:281)
	... 8 more
Caused by: azkaban.jobExecutor.utils.process.ProcessFailureException
	at azkaban.jobExecutor.utils.process.AzkabanProcess.run(AzkabanProcess.java:130)
	at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:296)
	... 9 more
11-04-2018 17:22:04 CST spark ERROR - java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException cause: java.lang.RuntimeException: azkaban.jobExecutor.utils.process.ProcessFailureException
11-04-2018 17:22:04 CST spark INFO - Finishing job spark at 1523438524471 with status FAILED

出现上述错误,在azkaban-exec-server-3.43.0/conf目录下修改azkaban.properties文件

在最后添加

azkaban.webserver.url=https://192.168.66.194:8666  #web加载页面
配置文件信息如下
#Azkaban
default.timezone.id=Asia/Shanghai

#Loader for projects
executor.global.properties=/root/azkaban/azkaban-exec-server-3.43.0/conf/global.properties
azkaban.project.dir=/root/azkaban/azkaban-exec-server-3.43.0/projects
azkaban.execution.dir=/root/azkaban/azkaban-exec-server-3.43.0/executions
project.temp.dir=/root/azkaban/azkaban-exec-server-3.43.0/temp

database.type=mysql
mysql.port=3306
mysql.host=192.168.66.170
mysql.database=azkaban
mysql.user=root
mysql.password=sql_9879
mysql.numconnections=100

#  Azkaban Executor settings

executor.maxThreads=50

executor.port=12321

executor.flow.threads=30

# JMX stats

jetty.connector.stats=true

executor.connector.stats=true

# Azkaban JobTypes Plugins
azkaban.jobtype.plugin.dir=/root/azkaban/azkaban-exec-server-3.43.0/plugins/jobtypes

# uncomment to enable inmemory stats for azkaban

#executor.metric.reports=true

#executor.metric.milisecinterval.default=60000

azkaban.webserver.url=https://192.168.66.194:8666


<think>好的,我现在要解决用户在Azkaban执行环境中遇到的ModuleNotFoundError: No module named 'pandas'的问题。首先,我需要理解这个错误的原因。通常情况下,这个错误意味着Python环境中没有安装pandas库。而用户提到可能与Azkaban有关,所以需要考虑Azkaban的工作流程和执行环境。 首先,我应该考虑用户的环境设置。Azkaban是一个工作流调度工具,通常用于在集群中运行作业。如果用户在Azkaban中运行Python脚本,那么需要确保执行任务的节点上已经安装了pandas库。这时候可能需要检查Azkaban的执行环境,确认Python环境是否正确配置,以及pandas是否被正确安装。 接下来,可能的解决方法包括: 1. **在Azkaban执行节点上安装pandas**:用户可能需要通过SSH登录到Azkaban的执行服务器,使用pip安装pandas。例如,运行`pip install pandas`或者如果使用特定版本的Python,可能需要用`pip3`。但需要注意权限问题,可能需要使用sudo或者安装在用户目录下。 2. **使用虚拟环境**:如果用户没有权限在全局安装库,可以创建一个虚拟环境,并在其中安装所需的依赖,然后在Azkaban作业配置中指定使用该虚拟环境中的Python解释器。 3. **打包依赖库**:将pandas库及其依赖打包到ZIP文件中,并在Python脚本中通过sys.path添加这个ZIP文件的路径。这样即使环境中没有安装pandas,也可以从ZIP中加载。 4. **使用Azkaban的依赖管理功能**:检查Azkaban是否支持指定依赖库,例如通过配置文件或作业属性来指定需要安装的Python包。 5. **检查Python路径和环境变量**:确保Azkaban作业配置中使用的Python路径正确,并且环境变量设置正确,特别是当有多个Python版本存在时。 在思考过程中,还需要考虑用户可能的具体情况。例如,用户是否拥有Azkaban服务器的访问权限?如果没有,可能需要其他方法,如打包依赖。此外,不同的环境(如Linux或Windows)可能安装方式不同,但Azkaban通常运行在Linux环境下,所以重点考虑Linux的安装方法。 另外,用户可能是在集群环境中运行,Azkaban可能将作业分发到多个节点执行,这时需要确保所有节点都安装了pandas库,而不仅仅是主节点。这种情况下,可能需要使用集群范围内的软件管理工具,或者在每个节点上预先安装好所需的库。 还有一种可能是用户的环境使用了Anaconda或Miniconda,这时可能需要用conda命令来安装pandas,例如`conda install pandas`,但同样需要考虑权限和环境激活的问题。 总结起来,解决方法大致分为三种:在服务器上安装库、使用虚拟环境或打包依赖。需要根据用户的具体环境和权限来选择最合适的方案。同时,要提醒用户测试安装后的环境是否正常工作,例如在Azkaban作业中运行简单的Python脚本验证pandas是否可导入。 最后,生成相关问题需要考虑用户可能遇到的后续问题,比如其他依赖库的安装、权限问题、虚拟环境的使用方法,或者Azkaban配置的调整等。</think>为了在Azkaban执行环境中解决`ModuleNotFoundError: No module named 'pandas'`错误,请按照以下步骤操作: ### 步骤1:确认执行环境类型 - 检查Azkaban作业配置中`command`字段使用的Python解释器路径(如`/usr/bin/python3`或虚拟环境路径) - 在本地终端执行`which python`或`which python3`验证Python环境一致性 ### 步骤2:服务器直接安装(需管理员权限) ```bash # 使用系统包管理器 sudo apt-get install python3-pandas # Ubuntu/Debian # 或 sudo yum install python3-pandas # CentOS/RHEL # 使用pip安装(推荐) pip install pandas --user # 用户级安装 # 或 sudo pip3 install pandas # 全局安装 ``` ### 步骤3:虚拟环境部署(无root权限时) ```bash # 创建虚拟环境 python3 -m venv azkaban_python_env # 激活环境并安装依赖 source azkaban_python_env/bin/activate pip install pandas deactivate # Azkaban作业配置示例 command: azkaban_python_env/bin/python your_script.py ``` ### 步骤4:依赖打包方案(集群环境适用) 1. 创建`requirements.txt`文件: ``` pandas==1.5.3 numpy==1.24.1 ``` 2. 打包依赖: ```bash pip install -t libs -r requirements.txt zip -r dependencies.zip libs/ ``` 3. 在Python脚本开头添加: ```python import sys sys.path.insert(0, 'dependencies.zip') ``` ### 步骤5:Azkaban工作流配置验证 ```properties # job.properties dependencies=dependencies.zip ``` ```python # 验证脚本test_pandas.py import pandas as pd print(f"Pandas version: {pd.__version__}") ``` ### 常见问题排查 - **权限问题**:使用`ls -l /usr/local/lib/python3.*/dist-packages`检查安装目录权限 - **路径冲突**:通过`import sys; print(sys.path)`查看模块搜索路径 - **版本兼容性**:检查Python版本是否符合pandas要求(需Python ≥3.7)
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值