青鱼的博客

SET mapreduce.job.queuename=yjy;SET hive.cli.print.header=TRUE;set hive.exec.dynamic.partition=true;  set hive.exec.dynamic.partition.mode=nonstrict;use dm_userimage;create table dm_userimage.f_u...

http://spark.apache.org/docs/latest/api/python/pyspark.sql.html参考官方文档from pyspark.sql import Windowcj_spouse_false = cj_spouse_false.withColumn("row_number", \                                    ...

前期操作参看的hdfs://cluster/user/hive/warehouse/dm_userimage.db/f_baseinfo_online_t有几点注意，1. 插入后查询会出现类型转换异常，es和hive 类型问题,建es对应表时候  int 改为bigint  date 改为timestamp  float改为double 2....

from pyhive import hiveconn = hive.Connection(host='1*.30',auth='LDAP',port=10000,username='*',password='*',database='*')cursor = conn.cursor()cursor.execute("select * from dp_ods.* where etl_date=...

脚本内部 spark = SparkSession.builder.master('yarn-client').config('spark.executor.instances', 10).config('spark.executor.cores',2). \        config('spark.executor.memory', '4g').config('spark.sql.sh...

base_info = spark.sql("select index, type,id, score, idcardinput,banknoteorremit,idcard,\        accounttype,accountopendate,nameinput,subaccount,usablebalance,querydate as querydate_base,balance as ...

stdi, stdo, stde = sys.stdin, sys.stdout, sys.stderrreload(sys)sys.setdefaultencoding('utf-8')sys.stdin, sys.stdout, sys.stderr = stdi, stdo, stde

/home/user_image/hadoop-2.7.2/bin/hadoop fs -mkdir hdfs://cluster/user/hive/warehouse/dm_userimage.db/f_userimage_messageinfo/etl_date=$yesterday/home/user_image/hadoop-2.7.2/bin/hadoop fs -cp /NS2/...

yf_u3.registerTempTable('yf_u3')spark.sql('DROP TABLE IF EXISTS dm_userimage.yf_u3')spark.sql('create table dm_userimage.yf_u3 select * from yf_u3')存入到指定分区INSERT OVERWRITE TABLE employeesPARTI...

load data inpath '/user/yjy_research/zhangyd/phonelist_result/2017-01-01' overwrite into table f_userimage_phonelist partition(etl_date='2017-01-01');

export PYSPARK_PYTHON=*/bin/python  $SPARK_HOME/bin/spark-submit --conf "spark.yarn.dist.archives=/home/anaconda2.tar.gz"  --conf  "spark.executorEnv.PYSPARK_PYTHON=/home/anaconda2.tar.gz/anaconda...