spark sql group by 出现问题

最新推荐文章于 2024-07-10 21:40:32 发布

time_exceed

最新推荐文章于 2024-07-10 21:40:32 发布

阅读量1.1w

点赞数 3

分类专栏： spark 文章标签： spark sql group-by

本文链接：https://blog.youkuaiyun.com/q5390498/article/details/53103840

版权

spark 专栏收录该内容

7 篇文章

订阅专栏

hiveContext.sql("select time,count(*) from page_click group by id").collect.foreach(println)
//报错
org.apache.spark.sql.AnalysisException: expression 'page_click.`time`' is neither present in the group by, nor is it an aggregate function. Add to group by or wrap in first() (or first_value) if you don't care which value you get.;
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:40)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:58)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.org$apache$spark$sql$catalyst$analysis$CheckAnalysis$class$$anonfun$$checkValidAggregateExpression$1(CheckAnalysis.scala:218)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$6.apply(CheckAnalysis.scala:245)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1$$anonfun$apply$6.apply(CheckAnalysis.scala:245)
  at scala.collection.immutable.List.foreach(List.scala:381)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:245)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$1.apply(CheckAnalysis.scala:67)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:67)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:58)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:582)
  at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:682)
  ... 48 elided

之后谷歌了一下，原来是time这一列在 group by的时候有多个查询结果，需要使用collect_set()一下。

scala>hiveContext.sql("select collect_set(time),count(*) from page_click group by id limit 10").collect.foreach(println)

OK!!!