Kafka+Streatured Streaming+Mysql项目实例

最新推荐文章于 2022-04-21 16:27:01 发布

zxucheng

最新推荐文章于 2022-04-21 16:27:01 发布

阅读量264

点赞数

CC 4.0 BY-SA版权

本文链接：https://blog.youkuaiyun.com/qq_27182767/article/details/86481500

本文介绍了一个实验项目，使用Kafka产生数据，Structured Streaming读取并处理数据，最后通过JdbcSink将结果写入Mysql数据库。实验环境包括Hadoop、Spark、Zookeeper、Kafka和mysql。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

一.实验内容

实验环境

Hadoop2.7.5

Spark2.4.0

Zookeeper -3.4.8

Kafka_2.11-2.1.0

mysql

实验内容

模拟产生数据写入Kafka,Structured Streaming读取Kafka中数据，通过Structured Streaming计算将数据写入Mysql,其中写入Mysql的过程中需要借助JdbcSink.

二.实验步骤

2.1 使用IDEA创建Maven项目

2,2生成模拟数据写入到kafka StockDataProducer.java

package com.processor.driver;
import java.util.Properties;
import java.util.Random;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;

public class StockDataProducer {
	private KafkaProducer<String, String> kafkaProducer;
	public static void main(String[] args) {
		StockDataProducer driver = new StockDataProducer();
		driver.run();
	}
	public void run() {
		Properties props = new Properties();

		props.put("bootstrap.servers", "192.168.100.10:9092");
		props.put("acks", "all");
		props.put("retries", 0);
		props.put("batch.size", 16384);
		props.put("linger.ms", 1);
		props.put("buffer.memory", 33554432);
		props.put("key.serializer",
				"org.apache.kafka.common.serialization.StringSerializer");
		props.put("value.serializer",
				"org.apache.kafka.common.serialization.StringSerializer");
		this.kafkaProducer = new KafkaProducer<String, String>(props);
		int i = 0;
		Random random = new Random();
		while (true) {
			for (i = 0; i <= 10; i++) {
				kafkaProducer.send(new ProducerRecord<String, String>(
						"market-data-input", "" + i,
						"1" + i + ",2019-01-14 15:35:45,ANZ"
								+ random.nextInt(10) + ",buyer,seller,"
								+ 100.0 * random.nextInt(100) + ","
								+ 10 * random.nextInt(100)));
			}
			try {