咸鱼的博客

本项目在腾讯云云服务器部署，部署完成之后默认打开，在之后配置的高可用，都没有写zookeeper的启动（因为已经打开了）。读者们请注意

#spark-env.sh中添加自动清理配置export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.interval=60 -Dspark.worker.cleanup.appDataTtl=120"

文章目录Spark Streaming实时写入HBase0--适用框架1-代码解析1-1 Kafka部分1-2 Hbase部分2-代码源码3-pom.xmlSpark Streaming实时写入HBase0–适用框架1-代码解析1-1 Kafka部分//定义一个主题数组，内可包含多个主题，此处只有一个val kafkaTopic = Array("mytopic")//bootstrap.servers kafka的服务节点和端口号，用于连接kafka//key.deserializer

爬虫代码Monitorpackage ln;import java.io.File;public class Monitor extends Thread{ @Override public void run() { super.run(); File file = new File("Data.txt"); while(true){ System.out.println("文件大小:"+file.lengt

Hadoop2.7.2 HA部署文章目录Hadoop2.7.2 HA部署解压改名配置环境变量配置 hadoop-env.sh配置yarn-env.sh配置mapred-env.sh配置slaves配置 core-site.xml配置 hdfs-site.xml配置yarn-site.xml初始化解压tar -zxvf hadoop-2.7.2.tar.gz -C /opt/改名cd /optmv hadoop-2.7.2 hadoop配置环境变量sudo vi /etc/profil

Linux安装JDK1.8.0解压tar -zxvf jdk-8u144-linux-x64.tar.gz -C /opt/改名cd /optmv jdk1.8.0_144/ jdk配置环境变量vim /etc/profile在末尾添加#JAVA_HOMEexport JAVA_HOME=/opt/jdkexport PATH=$PATH:$JAVA_HOME/bin保存退出:wq使环境变量生效source /etc/profile测试JDK是否安装成功ja

Kafka集群模式deploy解压安装tar -xzf kafka_2.13-2.8.0.tgz -C /optcd /opt/mv kafka_2.13-2.8.0 kafkacd kafkamkdir logscd configvim server.properties添加相关参数broker.id 唯一idlog.dirs 日志目录zookeeper.connect 使用外部zookeeper#

JAVA爬虫（爬取东方财富网）pom.xml<?xml version="1.0" encoding="UTF-8"?><project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http:/

Spark3.1.2高可用部署文档解压、改名tar -zxvf spark-3.1.2-bin-hadoop2.7.tgz -C /opt/cd /opt/mv spark-3.1.2-bin-hadoop2.7/ sparkcd spark/conf添加Hadoop配置文件的软链接ln -s /opt/hadoop/etc/hadoop/core-site.xmlln -s /opt/hadoop/etc/hadoop/hdfs-site.xml添加hive-site.xml配置文

CREATE TABLE hive_hbase_emp_table(empno int,ename string,job string,mgr int,hiredate string,sal double,comm double,deptno int)STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,info

Hive2.3.9部署解压安装改名tar -zxvf apache-hive-2.3.9-bin.tar.gz -C /optcd /opt/mv apache-hive-2.3.9-bin/ hivecd hivecd conf/mv hive-env.sh.template hive-env.shvim hive-env.sh添加以下内容export HADOOP_HOME=/opt/hadoopexport HIVE_CONF_DIR=/opt/hive/conf在hado

MySQL5.7部署安装linux CentOS7.9下载安装Yum镜像sudo rpm -Uvh mysql80-community-release-el7-3.noarch.rpm查看yum列表（ALL）yum repolist all | grep mysql禁用mysql80、启用mysql57sudo yum-config-manager --disable mysql80-communitysudo yum-config-manager --enable mysql57-co

Flume对接Kafka测试配置文件# example.conf: A single-node Flume configuration# Name the components on this agenta1.sources = r1a1.sinks = k1a1.channels = c1# Describe/configure the sourcea1.sources.r1.type = netcata1.sources.r1.bind = localhosta1.sources

flume安装使用安装tar -zxvf apache-flume-1.9.0-bin.tar.gz -C /opt/cd /opt/mv apache-flume-1.9.0-bin flumecd /opt/flume/confvim log4j.properties更改日志目录flume.log.dir=/opt/flume/logs运行测试touch /opt/flume/conf/example.confvim /opt/flume/conf/example.conf

Zookeeper 部署安装解压（每个服务器都要安装）tar -zxvf zookeeper-3.4.10.tar.gz -C /opt改名cd /optmv zookeeper-3.4.10 zookeeper修改配置文件cd /opt/zookeeper/confmv zoo_sample.cfg zoo.cfgvim zoo.cfg添加如下内容dataDir=/opt/zookeeper/zkData#######################cluster##

#!/bin/bash#1 获取输入参数个数，如果没有参数，直接退出pcount=$#if((pcount==0)); thenecho no args;exit;fi#2 获取文件名称p1=$1fname=`basename $p1`echo fname=$fname#3 获取上级目录到绝对路径pdir=`cd -P $(dirname $p1); pwd`echo pdir=$pdir#4 获取当前用户名称user=`whoami`#5 循环for((host=1; h