idea-spark-消除INFO日志内容

最新推荐文章于 2024-03-31 15:53:40 发布

原创最新推荐文章于 2024-03-31 15:53:40 发布 · 684 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#spark #intellij-idea #log4j

spark 专栏收录该内容

2 篇文章

订阅专栏

这篇博客介绍了如何在资源文件夹下添加log4j.properties文件来配置日志级别，包括设置日志输出到控制台，以及针对不同模块如Spark Shell、Parquet等的特定日志级别调整，旨在控制和优化应用的日志输出。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

只需在 resource 文件夹下添加 log4j.properties 文件，

并写入：

# Set everything to be logged to the console
log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR

# Parquet related logging
log4j.logger.org.apache.parquet.CorruptStatistics=ERROR
log4j.logger.parquet.CorruptStatistics=ERROR