<?xml version="1.0" encoding="utf-8" ?><rss version="2.0"><channel><title><![CDATA[WuBoooo的博客]]></title><description><![CDATA[]]></description><link>https://blog.youkuaiyun.com/WuBoooo</link><language>zh-cn</language><generator>https://blog.youkuaiyun.com/</generator><copyright><![CDATA[Copyright &copy; WuBoooo]]></copyright><item><title><![CDATA[spark 读 http 的 API 接口，将读出的JSON字符串，直接转成 DataFrame]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/132225055</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/132225055</guid><author>WuBoooo</author><pubDate>Fri, 11 Aug 2023 10:04:24 +0800</pubDate><description><![CDATA[【代码】spark 读 http 的 API 接口，将读出的JSON字符串，直接转成 DataFrame。]]></description><category></category></item><item><title><![CDATA[实测解决 flink cdc mysql 时间字段差8小时/差13小时问题]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/127387144</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/127387144</guid><author>WuBoooo</author><pubDate>Tue, 18 Oct 2022 14:08:39 +0800</pubDate><description><![CDATA[实测解决 flink cdc mysql 时间字段差8小时/差13小时问题]]></description><category></category></item><item><title><![CDATA[gp 索引列表查询，表字段类型，字段注释查询]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/123373625</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/123373625</guid><author>WuBoooo</author><pubDate>Wed, 09 Mar 2022 11:40:15 +0800</pubDate><description><![CDATA[--所有索引列表
SELECT n.nspname as "Schema", c.relname as "Name", c2.relname as "Table"
FROM pg_catalog.pg_class c
         JOIN pg_catalog.pg_roles r ON r.oid = c.relowner
         LEFT JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
         LEFT JOIN]]></description><category></category></item><item><title><![CDATA[Mysql binlog日志读出来的时间差8小时问题解决方案]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/122963383</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/122963383</guid><author>WuBoooo</author><pubDate>Wed, 16 Feb 2022 14:49:37 +0800</pubDate><description><![CDATA[//timeStamp = 2022-02-16T12:27:20.828524Z
            String zonedDateTime = LocalDateTime.parse(timeStamp.replace("Z", ""))
                    .atZone(ZoneId.of("UTC"))//设置老时区
                    .withZoneSameInstant(ZoneId.of("Asia/Shanghai"))//设置新时区
  ]]></description><category></category></item><item><title><![CDATA[flink 自定义带超时时间的计数窗口触发器]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/122881808</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/122881808</guid><author>WuBoooo</author><pubDate>Fri, 11 Feb 2022 15:40:07 +0800</pubDate><description><![CDATA[自定义带超时时间的计数窗口触发器
import org.apache.flink.api.common.functions.ReduceFunction;
import org.apache.flink.api.common.state.ReducingState;
import org.apache.flink.api.common.state.ReducingStateDescriptor;
import org.apache.flink.api.common.typeutils.base.LongSe]]></description><category></category></item><item><title><![CDATA[yarn的WEB页面上不能看log日志]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/122219818</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/122219818</guid><author>WuBoooo</author><pubDate>Wed, 29 Dec 2021 16:52:19 +0800</pubDate><description><![CDATA[要是yarn的WEB页面上不能看。 就直接 yarn logs -applicationId applicationid_XXX_XX    &gt; XXX_XX.txt

]]></description><category></category></item><item><title><![CDATA[greenplum主键存在就不插入的SQL]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/122080539</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/122080539</guid><author>WuBoooo</author><pubDate>Wed, 22 Dec 2021 10:51:36 +0800</pubDate><description><![CDATA[该sql当id=1的数据已存在时，就不会插入数据，如果不存在就正常插入
INSERT INTO ods.ods_iot_project
(id, project_id, status) SELECT 1,456,456 FROM (SELECT 1) tmp WHERE NOT EXISTS (
    SELECT 1 FROM ods.ods_iot_project WHERE id=1
    );


...]]></description><category></category></item><item><title><![CDATA[flink 通过事务和checkpoint的共同提交，保证EXACTLY_ONCE]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/121379765</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/121379765</guid><author>WuBoooo</author><pubDate>Wed, 17 Nov 2021 15:37:13 +0800</pubDate><description><![CDATA[获得gp的数据库连接池，使用连接池，不要使用单连接
package com.ysservice.dataStreamApi.utils;
import com.alibaba.druid.pool.DruidDataSourceFactory;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Properties;
/**
 * @Descripti]]></description><category></category></item><item><title><![CDATA[flink使用countWindow，往gp中批量写数据，并自定义超时触发器]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/121336158</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/121336158</guid><author>WuBoooo</author><pubDate>Mon, 15 Nov 2021 15:48:03 +0800</pubDate><description><![CDATA[设置超时触发时间类型为ProcessingTime
        env.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime);

主流程代码
dataStream
    .keyBy(r -&gt; r.f1)//按表名进行keyBy，避免同一张表的数据传入出现先后问题
    .timeWindow(Time.of(10, TimeUnit.SECONDS))//超时时间10秒
    .trigger(new CountTri]]></description><category></category></item><item><title><![CDATA[flink jdbc写gp，以多value的形式，快速写入]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/121294750</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/121294750</guid><author>WuBoooo</author><pubDate>Fri, 12 Nov 2021 19:53:06 +0800</pubDate><description><![CDATA[package com.ysservice.dataStreamApi.sink;

import com.ysservice.dataStreamApi.utils.GreenplumUtil;
import com.ysservice.dataStreamApi.utils.RegexUtils;
import org.apache.flink.api.java.tuple.Tuple3;
import org.apache.flink.configuration.Configuration;
impo]]></description><category></category></item><item><title><![CDATA[flink通过checkpoint的完成信号，提交jdbc sql，保证数据一致性]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/121268746</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/121268746</guid><author>WuBoooo</author><pubDate>Thu, 11 Nov 2021 15:27:19 +0800</pubDate><description><![CDATA[package com.ysservice.dataStreamApi.sink;

import com.ysservice.dataStreamApi.utils.GreenplumUtil;
import com.ysservice.dataStreamApi.utils.RegexUtils;
import org.apache.flink.api.common.state.CheckpointListener;
import org.apache.flink.api.java.tuple.Tupl]]></description><category></category></item><item><title><![CDATA[flink cdc 2.0.0 sql 开发模板，及踩坑记录]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/121036556</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/121036556</guid><author>WuBoooo</author><pubDate>Fri, 29 Oct 2021 15:56:02 +0800</pubDate><description><![CDATA[flink cdc sql 开发模板
flink cdc sql  读mysql的binlog日志，实时同步到mysql开发模板
使用flink cdc前提条件：读取目标库的用户必须开启binlog权限
package com.ysservice;
 
import com.typesafe.config.Config;
import com.typesafe.config.ConfigFactory;
import com.ysservice.utils.MyCheckpoint;
import com.]]></description><category></category></item><item><title><![CDATA[flink cdc 将binlog日志反序列化成sql语句]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/120830167</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/120830167</guid><author>WuBoooo</author><pubDate>Mon, 18 Oct 2021 17:13:37 +0800</pubDate><description><![CDATA[package com.ysservice.streaming.source;

import com.alibaba.ververica.cdc.debezium.DebeziumDeserializationSchema;
import com.ysservice.streaming.utils.MysqlUtil;
import com.ysservice.streaming.utils.SystemConstants;
import io.debezium.data.Envelope;
import]]></description><category></category></item><item><title><![CDATA[flink table api 有界无界窗口的使用]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/119816677</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/119816677</guid><author>WuBoooo</author><pubDate>Sat, 21 Aug 2021 00:17:06 +0800</pubDate><description><![CDATA[Table aggVin = inputTable.window(Over.partitionBy("vin").orderBy("rt").preceding("1.rows").as("w"))
                .select("vin,rt,carTime,min(carTime) over w as min_carTime")
                .select("vin,rt,carTime,min_carTime,(carTime - min_carT...]]></description><category></category></item><item><title><![CDATA[搜索ES内的数据]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/110475646</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/110475646</guid><author>WuBoooo</author><pubDate>Wed, 02 Dec 2020 11:33:02 +0800</pubDate><description><![CDATA[GET /charge_slice_computer/_search
{
  "sort": [
    {
      "charge_start_time": {
        "order": "desc"
      }
    }
  ]
}


]]></description><category></category></item><item><title><![CDATA[配置文件获取方法]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/110387033</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/110387033</guid><author>WuBoooo</author><pubDate>Mon, 30 Nov 2020 14:05:51 +0800</pubDate><description><![CDATA[private val config = ConfigFactory.load()
  lazy val cassConfig = config.getConfig("mysql")

  lazy val mysqlUrl = cassConfig.getString("mysql.url")
  lazy val username = cassConfig.getString("mysql.username")
  lazy val password = cassConfig.getString("m.]]></description><category></category></item><item><title><![CDATA[统一列的字段类型为:StringType]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/110386553</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/110386553</guid><author>WuBoooo</author><pubDate>Mon, 30 Nov 2020 13:53:33 +0800</pubDate><description><![CDATA[bigDF.select(bigDF.columns.map(column =&gt; {
      bigDF.col(column).cast(StringType)
    }): _*).na.fill("")


]]></description><category></category></item><item><title><![CDATA[Spark on Yarn详解]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/109479311</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/109479311</guid><author>WuBoooo</author><pubDate>Tue, 03 Nov 2020 21:18:01 +0800</pubDate><description><![CDATA[Spark on Yarn详解
Spark 可以跑在很多集群上，比如跑在local上，跑在Standalone上，跑在Apache Mesos上，跑在Hadoop YARN上等等。不管你Spark跑在什么上面，它的代码都是一样的，区别只是–master的时候不一样。其中Spark on YARN是工作中或生产上用的非常多的一种运行模式。今天主要对Spark on Yarn 这种方式做讲解。
yarn模式两种提交任务方式
Spark可以和Yarn整合，将Application提交到Yarn上运行，Yarn有两]]></description><category></category></item><item><title><![CDATA[spark-streaming_实时写入hbase]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/109457108</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/109457108</guid><author>WuBoooo</author><pubDate>Mon, 02 Nov 2020 21:13:05 +0800</pubDate><description><![CDATA[当处理实时数据是聚合类的运算是,可以写入到mysql中,因为数据量不大,但如果是非聚合类的数据,mysql中存放不下,此时可以使用支持覆盖写入或事务的大型数据库,例如:hbase,ES,clickhouse
hbase在写入数据时如果行键相同的数据写进来就会覆盖原始数据,所以当我们在运算时将每条数据赋予唯一的行键(例如:订单号,或者设备号加时间戳),即使一批数据写入到一半时中断了,重新写入时会覆盖之前的值,保证了数据的幂等性
在写入数据时,记偏移量,可以将偏移量和数据记在同一行不同的列族中(hbase支持行]]></description><category></category></item><item><title><![CDATA[redis介绍]]></title><link>https://blog.youkuaiyun.com/WuBoooo/article/details/109456489</link><guid>https://blog.youkuaiyun.com/WuBoooo/article/details/109456489</guid><author>WuBoooo</author><pubDate>Mon, 02 Nov 2020 20:35:12 +0800</pubDate><description><![CDATA[Redis
Nosql技术
1redis介绍
1.1什么是NoSql
NoSql是为了解决高并发、高可扩展、高可用以及高写入而产生的数据库解决方案。
NoSql就是Not Only sql。Nosql是非关系型数据库，它是关系型数据库的良好补充，而不能替代关系型数据库。
1.2Nosql数据库分类(了解)
键值(Key-Value)存储数据库
相关产品： Tokyo Cabinet/Tyrant、Redis、Voldemort、Berkeley DB
典型应用： 内容缓存，主要用于处理大量数据的高访问负载]]></description><category></category></item></channel></rss>