qq_38680817-优快云博客

原创 SparkUtils工具类

package com.xxx.sparktest.utilsimport java.sql.DriverManagerimport java.util.concurrent.LinkedBlockingQueueimport java.util.concurrent.atomic.AtomicIntegerimport java.util.{Calendar, Map, Properties}import com.alibaba.fastjson.{JSONArray, JSONObjec.

2022-01-17 15:46:49 444

原创解决写入mysql utf8 无法识别4字符编码

通常情况，Mysql数据编码格式为“utf-8”，对于汉字来说足够；Mysql中utf8占3个字节，但是，3个字节对于表情符号是不够的，需4个字节；此时使用utf8，会出现‘\xF0\x9F\x8D\x83\xF0\x9F’的问题。1、异常java.sql.BatchUpdateException: Incorrect string value: '\xF0\x9F\x8D\x83\xF0\x9F...' for column 'nickname' at row 1通常情况，Mysql数据编..

2022-01-10 16:20:10 613

原创阿拉伯转数字

package com.sf.gis.utils;import java.util.ArrayList;import java.util.List;import java.util.Stack;import java.util.regex.Matcher;import java.util.regex.Pattern;public class StringNumUtils { public static int transferChineseNumber2ArabNumber(Str..

2022-01-10 16:14:13 444

原创 sparksql源码解析一：

parser调用sqlbaseParser的singleStatement方法，利用anltr4里面的singleStatement来目前主流的sql解析器有 anltr和calcite，如果使用选择使用anltr，SQL进行查询，首先需要将SQL解析成spark中的抽象语法树（AST）。在spark中是借助开源的antlr4库来解析的。Spark SQL的语法规则文件是：SqlBase.g4。在生成的文件中SqlBaseBaseListene

2021-12-07 10:09:21 1409

原创 maven conf

<?xml version="1.0" encoding="UTF-8"?><!--Licensed to the Apache Software Foundation (ASF) under oneor more contributor license agreements. See the NOTICE filedistributed with this work for additional informationregarding copyright ownershi.

2021-11-30 11:38:02 186

原创 hive修改压缩方式

ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'WITH SERDEPROPERTIES ('field.delim'='\t','line.delim'='\n','serialization.format'='\t')STORED AS PARQUET TBLPROPERTIES('parquet.compression'='SNAPPY')ALTER TABLE dm_gis..

2021-10-13 16:48:45 1206

原创 hive和sparksql对数组格式日志处理

sparksqlregexp_replace( regexp_extract(tracks2, '\\\\[\\\\[[^\\\\]]+', 0), '\\\\[\\\\[', '' ) as starts, regexp_replace( regexp_extract(tracks2, '[^\\\\[]+\\\\]\\\\]', 0), ...

2021-09-28 11:39:17 136

原创多张报表简单逻辑在同一任务中样例

import java.text.SimpleDateFormatimport java.utilimport java.util.Dateimport com.alibaba.fastjson.JSONObjectimport com.fengtu.sparktest.utils.{JSONUtils, MD5Util, SparkUtils}import com.fengtu.sparktest.utils2.DateUt.

2021-08-02 10:26:26 224

原创 shell执行工具类

import org.slf4j.Logger;import org.slf4j.LoggerFactory;import java.io.IOException;public class ShellExcutor { private static Logger logger = LoggerFactory.getLogger(ShellExcutor.class); public static void e.

2021-08-02 10:19:15 182

原创 spark工具类（常用）

package com.fengtu.sparktest.utilsimport java.sql.DriverManagerimport java.util.{Map, Properties}import com.alibaba.fastjson.{JSONArray, JSONObject}import com.fengtu.sparktest.utils2.Utilsimport org.apache.log4j.Loggerimport org.apache.spark.rdd.R.

2021-08-02 10:18:11 586

原创时间日期工具类

package com.fengtu.sparktest.utilsimport java.text.{ParseException, SimpleDateFormat}import java.util.{Calendar, Date}import scala.collection.mutable.ArrayBufferobject DateTimeUtil extends Serializable { def main(args: Array[String]): Unit = {.

2021-08-02 10:17:10 111

原创 spark多线程的案例

package com.fengtu.sparktest.aoiTestimport java.text.SimpleDateFormatimport java.util.Dateimport java.util.concurrent.{Callable, Executors, FutureTask}import com.alibaba.fastjson.{JSON, JSONArray, JSONObject}import com.fengtu.sparktest.aoi.GetGjAoi.

2021-08-02 10:15:08 516

原创 spark写入mysql业务案例

package com.fengtu.sparktest.aoiTestimport java.sql.Connectionimport java.security.MessageDigestimport com.alibaba.fastjson.JSONObjectimport com.fengtu.sparktest.utils.{DbUtils, JavaUtil, JdbcTemplateUtil, StringUtils}import org.apache.log4j.Logger.

2021-08-02 10:13:30 132

原创 spark业务案例

package com.fengtu.sparktest.aoiTestimport java.net.URLEncoderimport com.alibaba.fastjson.serializer.SerializerFeatureimport com.alibaba.fastjson.{JSON, JSONArray, JSONObject}import com.fengtu.sparktest.utils.SparkUtilsimport com.fengtu.sparktest.u.

2021-08-02 10:12:08 307

原创 spark复杂转换

package com.sf.gis.scala.base.sparkimport com.alibaba.fastjson.JSONObjectimport org.apache.log4j.Loggerimport org.apache.spark.rdd.RDDimport scala.collection.mutable.ArrayBufferimport scala.util.Random/** * Created by 01374443 on 2020/7/27. .

2021-08-02 10:10:47 129

原创 spark网络调优

package com.sf.gis.scala.base.sparkimport java.util.Calendarimport java.util.concurrent.{Executors, LinkedBlockingQueue}import java.util.concurrent.atomic.AtomicIntegerimport com.alibaba.fastjson.JSONObjectimport org.apache.log4j.Loggerimport org..

2021-08-02 10:10:16 164

原创 spark读工具类

package com.sf.gis.scala.base.sparkimport org.apache.log4j.Loggerimport org.apache.spark.rdd.RDDimport org.apache.spark.sql.{Row, SparkSession}import org.apache.spark.storage.StorageLevelimport com.alibaba.fastjson.JSONObjectimport com.sf.gis.java..

2021-08-02 10:09:37 166

原创 spark写入工具类

package com.sf.gis.scala.base.sparkimport java.util.Propertiesimport com.alibaba.fastjson.JSONObjectimport com.sf.gis.java.base.dto.DBInfoimport com.sf.gis.java.base.pojo.BasePojoimport com.sf.gis.java.base.util.{CalPartitionUtil, ObjectUtil}impor.

2021-08-02 10:09:01 157

原创 Spark初始化工具类

package com.sf.gis.scala.base.sparkimport org.apache.spark.SparkConfimport org.apache.spark.sql.SparkSessionimport org.apache.log4j.Logger/** * Created by 01374443 on 2020/7/27. */object Spark { @transient lazy val logger: Logger = Logger.get.

2021-08-02 10:07:13 259 1

原创 spark处理复杂情况的join

package com.sf.gis.scala.base.sparkimport org.apache.log4j.Loggerimport org.apache.spark.rdd.RDDimport scala.collection.mutable.ArrayBufferimport scala.util.Random/** * Created by 01374443 on 2020/7/27. * 处理一些复杂的关联场景 */object SparkJoin { .

2021-08-02 10:03:58 179

原创 20210716重排链表

题目描述给定一个单链表L: L0→L1→…→Ln-1→Ln,重新排列后为：L0→Ln→L1→Ln-1→L2→Ln-2→…必须在不改变节点值的情况下进行原地操作。样例说明样例 1: 输入: 1->2->3->4->null 输出: 1->4->2->3->null样例 2: 输入: 1->2->3->4->5->null 输出: 1->5->2->4->...

2021-07-16 10:41:00 72

原创链表刷题题目

必刷题目：1.合并k 个排序链表2.带环链表3.翻转链表4.两个链表的交叉5.链表划分6.重排链表7.回文链表8.加一链表9.奇偶链表10.两数相乘

2021-07-13 14:14:00 58

原创字符串刷题

1.字符串查找2.最长回文子串3.有效回文串4.单词拆分I5.正则表达式匹配6.旋转字符串7.解码方法8.至少K 个不同字符的子串9.字符串查找II10.全零子串的数量11.最长公共子序列12.编辑距离13.第一个只出现一次的字符14.乱序字符串15.Excel 表列标题16.一次编辑距离17.删除无效的括号18.单词的添加与查找19.基础计算器20.比较字符串...

2021-07-13 14:12:15 92

原创 2021-06-25

import java.text.SimpleDateFormatimport java.util.Dateimport java.util.concurrent.{Callable, Executors, FutureTask}import com.alibaba.fastjson.{JSON, JSONArray, JSONObject}import com.fengtu.sparktest.aoi.GetGjAoiInterface2import org.apache.commons...

2021-06-25 16:26:54 193

原创 2021-05-31

/** * * @param spark * @param rdd * @param schema * @param user * @param password * @param saveMode * @param jdbcUrl * @param tblName * @param incDay * @param logger * @param statdate */def df2Mysql(spark: SparkSession,rdd: RDD[.

2021-05-31 10:57:43 50

原创 2021-05-31

package com.fengtu.sparktest.utilsimport java.io.{BufferedReader, IOException, InputStream, InputStreamReader}import java.sql.{Connection, PreparedStatement, SQLException}import java.util.Propertiesimport com.alibaba.druid.pool.DruidDataSourceimpor.

2021-05-31 10:56:44 64