球球呀。-优快云博客

原创 kafka集群脚本启动

#! /bin/bash case $1 in “start”){ for i in vm0 vm1 vm2 do echo " --------启动 $i Kafka-------" # 用于KafkaManager监控 ssh $i “export JMX_PORT=9988 && /opt/software/kafka_2.11-0.11.0.0/bin/kafka-server-start.sh -daemon /opt/software/kafka_2.11-0.11.0.0/co

2020-07-26 22:32:38 234

原创 flume脚本启动

#! /bin/bash case $1 in “start”){ for i in vm2 do echo " --------启动 $i 消费flume-------" ssh $i “nohup /opt/software/flume1.8/bin/flume-ng agent --conf-file /opt/software/flume1.8/conf/kafka-flume-hdfs.conf --name a1 -Dflume.root.logger=INFO,LOG FILE >/op

2020-07-26 22:29:41 740

原创 scala生成数据使用kafka发送并保存在本地

import java.io.PrintWriter import java.text.SimpleDateFormat import java.util.{Date, Properties} import org.apache.kafka.clients.producer.{KafkaProducer, ProducerRecord} import org.apache.kafka.common.serialization.StringSerializer import scala.util.Rand

2020-06-09 22:16:36 602

原创 scala模拟数据利用kafka输出并写入本地，并在本地进行在线计算

/** 模拟数据数据格式如下：日期卡口ID 摄像头编号车牌号拍摄时间车速道路ID 区域ID date monitor_id camera_id car action_time speed road_id area_id monitor_flow_action monitor_camera_info @author Administrator / object Mock

2020-06-08 20:15:16 732

原创 sparksql使用hive数据源保存在mysql数据库

//导入包并配置 hive数据源 val spark = SparkSession.builder() .master(“local”) .appName(“AreaTopNRoadFlowAnalyzer”) .config(“fs.defaultFS”, “hdfs://ip:9000”) //数据存放位置 .config(“spark.sql.warehouse.dir”, “hdfs://ip:9000/test”) .enableHiveSupport() .getOrCreate() spark

2020-06-07 21:28:25 421

原创 sparkStreaming整合kafka生产者实现单词统计

开启zookeeper，kafka，生产者， [qiushengfei@qiushengfei kafka]$ bin/kafka-console-producer.sh --broker-list ip:9092 --topic 主题名 //创建sparkConf val conf: SparkConf = new SparkConf().setAppName("kafka_receiver").setMaster("local[4]") //创建sparkcontext val sc: SparkCon

2020-06-07 19:07:44 486

原创 sparkStreaming整合flume实现单词统计推模式Push

** sparkStreaming整合flume 推模式Push spark-streaming-flume-sink_2.11-2.0.2.jar放入到flume的lib目录下 flume-push.conf #push mode a1.sources = r1 a1.sinks = k1 a1.channels = c1 #source a1.sources.r1.channels = c1 a1.sources.r1.type = spooldir a1.sources.r1.spoolDir =

2020-06-07 10:47:02 181

原创 sparkStreaming整合flume实现单词统计 poll模式

** sparkStreaming整合flume 拉模式Poll 配置文件 flume-poll.conf a1.sources = r1 a1.sinks = k1 a1.channels = c1 #source a1.sources.r1.channels = c1 a1.sources.r1.type = spooldir a1.sources.r1.spoolDir = /opt/flume-1.7.0/data a1.sources.r1.fileHeader = true #channel

2020-06-07 10:42:47 191

原创批量删除的方式

var ids=$("[name=‘ck’]:checked").map(function(){ return this.value; }).get().join(); 即可选中选中的多选框然后用ajax传入后台，用string类型接受

2020-01-15 14:41:20 150

qq_39759106的博客