m0_37135879的博客

Flink Table&Sql使用问题

集群规划我安装之前其他软件已经正常的配置,本教程只介绍Kylin安装配置解压包[root@Linux122 servers]# tar -zxvf apache-kylin-3.1.1-bin-hbase1x.tar.gz环境变量添加#Kylinexport KYLIN_HOME=/opt/lagou/servers/kylin-3.1.1export PATH=$PATH:$KYLIN_HOM/bin软连接添加其他组件配置ln -s /opt/lagou/servers/hadoo

由于Kudu对集群时间精准因此需要安装ntp服务一.安装ntp服务[root@Linux121 clickhouse-server]# yum -y install ntp[root@Linux121 clickhouse-server]# vim /etc/ntp.conf 各个节点注释下边几行#server 0.centos.pool.ntp.org iburst#server 1.centos.pool.ntp.org iburst#server 2.centos.pool.ntp.o

一.安装官网介绍安装https://clickhouse.com/#quick-startsudo yum install -y yum-utilssudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.reposudo yum install -y clickhouse-server clickhouse-clientsudo /etc/init.d/clickhouse-server

从官网下载安装包一. Standalone模式https://flink.apache.org/downloads.html#all-stable-releases[root@Linux121 conf]# vim flink-conf.yaml修改主master节点jobmanager.rpc.address: Linux121修改主从节点信息[root@Linux121 conf]# cat mastersLinux121:8081[root@Linux121 conf]# cat w

这两天在学习join的几种方式,其中提到hash等值和不等值连接,于是进行了概念清晰首先我先说下我第一误区,我等值和不等值难道是只有数字相等吗？不包括字符串其他吗？于是对hash值进行重新研究https://zhuanlan.zhihu.com/p/118148884通过这篇文章我们更进一步了解到hash其实可以针对多种数据类型数据是会有各种格式的，字符串，浮点……那么第二个疑问我们在SparkSql中指的不等值连接是什么？其实是类似 >,< bettwen …and…等操作,有种

一.行转列操作import org.apache.spark.rdd.RDDimport org.apache.spark.sql.{Dataset, Row, SparkSession}case class Info(id:String,tags:String)object SQLDemo { def main(args: Array[String]): Unit = { val spark: SparkSession = SparkSession.builder()

一.WordCount数据hadoop mapreduce yarnhdfs hadoop mapreducemapreduce yarn lagoulagoulagou案例:Scala版def main(args: Array[String]): Unit = { val conf = new SparkConf() conf.setAppName(this.getClass.getCanonicalName) conf.setMaster("local[*]") va

背景：集群部署几种模式集群模式一.Standalone模式1.解压[root@Linux121 servers]# tar -zxvf spark-2.4.5-bin-without-hadoop-scala-2.12.tgz2.配置环境变量export SPARK_HOME=/opt/lagou/servers/spark-2.4.5export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin3.修改配置文件[root@Linux121 sp

一.Spark安装有二进制包,不用编译tar -zxvf spark-2.2.1-bin-hadoop2.7.tgz设置环境变量export SPARK_HOME=/opt/lagou/servers/spark-2.2.1export PATH=$PATH:$SPARK_HOME/bin修改配置文件/opt/lagou/servers/spark-2.2.1/conf/spark-defaults.conf增加如下配置spark.master yarn

一.解压二.修改pom文件 645 <npm-for-v2.version>3.10.8</npm-for-v2.version> 652 <hadoop.version>2.9.2</hadoop.version>三.创建文件夹,手动放入solr,hbase(不然maven下载时间较长)/opt/lagou/servers/apache-atlas-sources-1.2.0/distro[root@Linux1

一.卸载 mariadb(卸载和mysql冲突的分支)[root@Linux121 guchenfei]# rpm -qa | grep mariadbmariadb-libs-5.5.68-1.el7.x86_64[root@Linux121 guchenfei]# yum remove mariadb-libs //根据查询结果remove对应的依赖包.......Removed: mariadb-libs.x86_64 1:5.5.68-1.el7

一.数据准备下表为分区增量数据表(userinfo)创建表加载数据用户信息create table test.userinfo( userid string comment '用户编号', mobile string comment '手机号码', regdate string comment '注册日期')comment '用户信息'partitioned by(dt string)row format delimited fields terminated by ','

1.解压压缩包tar -zxvf apache-tez-0.9.2-bin.tar.gz2.将tez路径下的压缩包放到hdfs上/opt/lagou/servers/tez/share/tez.tar.gzhdfs dfs -mkdir /user/tezhdfs dfs -put tez.tar.gz /user/tez/3.配置hadoop让其他地方知道该路径该路径下新建配置文件tez-site.xml/opt/lagou/servers/hadoop-2.10.1/etc/had

1，vgdisplay 得到vg Name 为centos[root@Linux122 guchenfei]# vgdisplay --- Volume group --- VG Name centos System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 5 VG Access read

数据背景hive (ods)> select * from jsont1;OKjsont1.username jsont1.age jsont1.sex jsont1.jsonuser1 18 male {"id":1,"ids":[101,102,103],"total_number":3}user2 28 female {"id":2,"ids":[201,202,203],"total_number":3}user3 38 male {"id":3,"ids":[301,302,30

笔者意外发现自己想要的依赖没有打包到jar里,于是研究发现需要在pom文件中添加以下配置才能按照自己pom配置打包.不需要打包的依赖添加如下标记<scope>provided</scope>例如： <dependency> <groupId>org.apache.flume</groupId> <artifactId>flume-ng-core</artifactId>

lateral view 的语法格式lateral view:LATERAL VIEW udtf (expression) tableAlias AS coluumAlias ( ‘,’ , columnAlias)*fromClause:FROM baseTable (lateralView)*用法描述lateral view 要与UDTF函数一起使用，比如 explode() 函数。这里的UDTF函数是指用户自定义的表生成函数（英文全称：user defined table-generati

**一.准备**gradle:gradle-4.8.1kafka源码包 Kafka-1.0.2scala 2.13.7二.gradle环境搭建解压安装包配置环境变量#GRADLE_HOMEGRADLE_HOME=/Users/guchenfei/Source/gradle-4.8.1export PATH=$PATH:$GRADLE_HOME/bin验证gradle环境gradle -versionguchenfeideMacBook-Pro:~ guchenfei$ grad

一.开启Kafka监控开启JMX工具(Orcle提供)vim kafka-server-start.sh添加export JMX_PORT=9581启动kafka[root@Linux122 bin]# ss -nelp //查看所有网络端口信息[root@Linux122 bin]# ss -nelp | grep 9581u_str LISTEN 0 10 @/tmp/dbus-hhAEH1DO 29581 * 0

创建主题观察偏移量相关数据[root@Linux122 bin]# kafka-topics.sh --zookeeper Linux122:2181/myKafka --create --topic tp_demo_06 --partitions 5 --replication-factor 1WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide

关于Kafka常见操作1.查看zookeeper保存的消息主题kafka-topics.sh --zookeeper localhost:2181/myKafka --list2.生产者向broker生产消息kafka-console-producer.sh --broker-list 172.16.131.130:9000 --topic topic_1 (其中172.16.131.130:9000是因为我在config/server.properties下配置的listeners=PLAIN

一.生产者消费者1.生产者生产消息方式同步方式public class Myproducer1 { public static void main(String[] args) { Map<String, Object> configs = new HashMap<>(); configs.put("bootstrap.servers", "Linux122:9092"); configs.put("key.seriali

Hive中定义分割符会使用八进制的ASCII码问题描述：今天在用Azkaban跑job的时候发现出了如下问题14-11-2021 15:50:00 CST analysis INFO - MismatchedTokenException(24!=347)14-11-2021 15:50:00 CST analysis INFO - at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java: