lyw5200的博客

开发工具：IDEA开发环境：win10本地环境，spark运行spark程序时，控制台总会输出很多红色日志，默认日志输出级别为INFO，可通过以下两种方法进行日志级别的修改方法一：在代码块中进行日志级别的设置 val sc = SparkSession.builder() .config("spark.testing.memory","2000000000") .master("local[*]") .appName("aiwen.com")

centos-python3环境下，调用pyspark库连接mysql数据库时报错：java.sql.SQLException: No suitable driver具体如下： File "/usr/local/lib/python3.6/site-packages/pyspark/sql/readwriter.py", line 155, in load return self._df(self._jreader.load()) File "/usr/local/lib/pytho

已解决：Exception: Python in worker has different version 2.7 than that in driver 3.6, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.在阿里云服务器上运行pyspark模块程序时，核心报错如