阿里云提交spark 任务找不到 project，是因为endpoint不对

最新推荐文章于 2023-01-04 23:56:09 发布

poson

最新推荐文章于 2023-01-04 23:56:09 发布

阅读量680

点赞数

分类专栏：阿里云文章标签：阿里云 spark

本文链接：https://blog.youkuaiyun.com/poson/article/details/105847632

版权

阿里云专栏收录该内容

4 篇文章

订阅专栏

本文详细介绍了在阿里云上使用Spark提交任务时遇到的问题及解决方案，主要聚焦于因spark-odps的endpoint配置错误导致的任务提交失败。通过调整endpoint设置，成功解决了找不到project的问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

阿里云提交spark 任务找不到 project，发现是因为spark -odps 的endpoint 不正确，从阿里云project 的配置（https://setting-cn-beijing.data.aliyun.com/#/dataSource）中找到endpoint，然后修改endpoint 之后就正确了。

 ./bin/spark-submit --jars cupid/odps-spark-datasource_2.11-3.3.9.jar exp.py


20/04/29 17:04:27 INFO Utils: Copying /Users/xxxx/bin/spark-2.3.0-odps0.32.2/exp.py to /private/var/folders/yp/8602rk5x1xvc9vssg14gglzc0000gp/T/spark-3aad704c-6cad-47be-90f9-895e644af535/userFiles-c8a2d588-e0cd-41f4-b13c-e50c959c337f/exp.py

20/04/29 17:04:27 INFO Executor: Starting executor ID driver on host localhost

20/04/29 17:04:27 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52238.

20/04/29 17:04:27 INFO NettyBlockTransferService: Server created on 30.225.8.197:52238

20/04/29 17:04:27 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy

20/04/29 17:04:27 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 30.225.8.197, 52238, None)

20/04/29 17:04:27 INFO BlockManagerMasterEndpoint: Registering block manager 30.225.8.197:52238 with 366.3 MB RAM, BlockManagerId(driver, 30.225.8.197, 52238, None)

20/04/29 17:04:27 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 30.225.8.197, 52238, None)

20/04/29 17:04:27 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 30.225.8.197, 52238, None)

20/04/29 17:04:28 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/spark-warehouse').

20/04/29 17:04:28 INFO SharedState: Warehouse path is 'file:/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/spark-warehouse'.

20/04/29 17:04:28 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint

Traceback (most recent call last):

  File "/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/exp.py", line 5, in <module>

    spark.sql("CREATE TABLE spark_sql_test_table(name STRING, num BIGINT)")

  File "/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/python/lib/pyspark.zip/pyspark/sql/session.py", line 708, in sql

  File "/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__

  File "/Users/zhenghong/bin/spark-2.3.0-odps0.32.2/python/lib/pyspark.zip/pyspark/sql/utils.py", line 71, in deco

pyspark.sql.utils.AnalysisException: u"Database 'pai_rec_dev' not found;"

20/04/29 17:04:42 INFO SparkContext: Invoking stop() from shutdown hook

20/04/29 17:04:42 INFO SparkUI: Stopped Spark web UI at http://30.225.8.197:4040

20/04/29 17:04:42 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!