杨羊不是羊的博客

使用collect之后 要取map里第0个元素select("ad_id").collect().map(_(0)).toList

val weightList = Array(1 - validationRatio - testRatio, validationRatio, testRatio)val dsList = result.randomSplit(weightList, splitSeed)val dfList = dsList.map(_.toDF)val trainDF = dfList(0)val validDF = dfList(1)val testDF = dfList(2)如果用sql的row_nu

import org.apache.log4j.Loggerimport org.apache.log4j.LevelLogger.getLogger(“org”).setLevel(Level.OFF)Logger.getLogger(“akka”).setLevel(Level.OFF)

scala之拼接

scala窗口函数这排序rankimport org.apache.spark.sql.expressions.Windowimport spark.implicits._val testDF =Seq( ("A", 50), ("B", 39), ("A", 48), ("A", 48), ("B", 35), ("C", 42), ("C", 60), ("C", 45), ("C", 52), ("C", 52)).toDF("name","score"

import java.net.URIval putPath = new Path(modelPath)val conf = new Configuration()val hdfs = FileSystem.newInstance(URI.create(modelPath),conf)if (hdfs.exits(putPath)){println("1")}else{println("2")}if (hdfs.getFileStatus(putPath).isDirectory){pr

#rdd求差集val monthActiveImei = sql(""“select imei from ad_tag.f_tag_month_active_user where dayno = 20200512"”")val kuaishouYearImei = sql(""“select imei from ad_tmp.test_tag_video_0513_02_kuaishou”"")val monthActiveImeiRdd=monthActiveImei.rdd.map(x =>

Scala之udf1.两个list(String)做内积定义内积udf，两个list做内积，list不能直接toInt，需要map toInt。//定义内积udfdef getInner(listNameA:String,listNameB:String):Int={ val listIntA = listNameA.split(",").map(_.toInt) val listIn...