前言
平时经常性的有一些临时统计数据需求,用hive虽然很方便,但是等待时间有点长,spark-shell成为了我常用的一种方式,不过,一般我都是在IDEA把代码写好,然后复制到spark-shell上面,这个时候就会出现如下问题:
比如我复制如下代码(有换行):
sql(sqlText = "select statis_month,count(*) from ods.ods_app_score_m where statis_month " +
"like '2019%' group by statis_month")
.show()
spark-shell中竟然显示如下:
scala> sql(sqlText = "select statis_month,count(*) from ods.ods_app_score_m where statis_month " +
| "like '2019%' group by statis_month")
res0: org.apache.spark.sql.DataFrame = [statis_month: string, count(1): bigint]
scala> .show()
显然sql中的部分,它自动识别了换行,dataset的算子部分它并没有识别出来,就默认为是另一段代码了
如何解决
直接看操作
scala> :paste
// Entering paste mode (ctrl-D to finish)
sql(sqlText = "select statis_month,count(*) from ods.ods_app_score_m where statis_month " +
"like '2019%' group by statis_month")
.show()
// Exiting paste mode, now interpreting.
[Stage 0:> (0 + 0) / 16080]
解释:
- 输入 :paste ,进入spark-shell的复制模式
- 粘贴所复制的内容,敲回车键
- 按下ctrl + D开始执行代码
后记
spark-shell中有很多命令都是以:开始,比如退出为:quit,因此需要多多总结才能更加熟练使用spark-shell,可以输入:help查看常用的命令,如下:
scala> :help
All commands can be abbreviated, e.g., :he instead of :help.
:edit <id>|<line> edit history
:help [command] print this summary or command-specific help
:history [num] show the history (optional num is commands to show)
:h? <string> search the history
:imports [name name ...] show import history, identifying sources of names
:implicits [-v] show the implicits in scope
:javap <path|class> disassemble a file or class name
:line <id>|<line> place line(s) at the end of history
:load <path> interpret lines in a file
:paste [-raw] [path] enter paste mode or paste a file
:power enable power user mode
:quit exit the interpreter
:replay [options] reset the repl and replay all previous commands
:require <path> add a jar to the classpath
:reset [options] reset the repl to its initial state, forgetting all session entries
:save <path> save replayable session to a file
:sh <command line> run a shell command (result is implicitly => List[String])
:settings <options> update compiler options, if possible; see reset
:silent disable/enable automatic printing of results
:type [-v] <expr> display the type of an expression without evaluating it
:kind [-v] <expr> display the kind of expression's type
:warnings show the suppressed warnings from the most recent line which had any