flink系列2 wordcount测试(windows)

在 Windows 搭建 flink 运行环境并模拟流数据处理 在大数据场景中,开发者追求高效与灵活,Linux 系统以其稳定性成为众多组件的首选,但在资源有限的情况下,在本机搭建一个 Linux 虚拟机集群却显得过于笨重,启动、运行占资源,需要配置网络,无法和windows共享资源,尤其是对只有 8GB 内存的 Windows 系统用户来说,内存压力显而易见。尽管如此,也挡不住一个学习者的“热情”。本文介绍怎么在 Windows 本地搭建 Flink 环境,注意这是一个伪分布式的测试环境。 阅读详情

参考文档
https://ci.apache.org/projects/flink/flink-docs-master/quickstart/setup_quickstart.html

windows上测试,linux上2017年4月的存档中有
1、准备工具。在百度或者google上搜索netcat-win32-1.11.zip。
运行nc64 -l -p 9099。启动监听程序

2、安装java

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

3、下载最新的flink1.4.0版本
解压
D:\flink-1.4.0-bin-hadoop27-scala_2.11\flink-1.4.0

4、启动flink的本地模式


        
flink经典实战案例 一、java版flink-wordcount-离线计算版 1.1maven构建flink,加入依赖 <!-- flink包依赖配置-start --> <dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-java</artifactId> <version>${flink.version 阅读详情

相关推荐

windows、linux下flink初体验

最近在看尚学堂的大数据,学习了scala,阅读大数据相关的公众号了解了flink,忍不住要体验一把。 下载 flink下载可以直接到官网下载,选择合适的镜像网站下载即可,速度很快,我两分钟就下载完了 如果使用scala进行开发,下载的时候,注意选择对应的版本 我本地安装的是Java 1.8.0_201 / Scala 2.11.11 安装 windows安装 window...

远方·玲玲 730

【Flink】第二节 windows下运行

接上一节,flink编译ok了,但是我准备在windows上进行运行,但是好像1.10之后 start_cluster.bat没有了呢,这样我怎么运行呢,本节是准备这个问题。

Chain的博客 1805

Flink 1.20单机部署指南:Docker环境配置与实战

流处理引擎Apache Flink作为实时计算领域的核心技术,其容器化部署已成为现代大数据架构的标配。Docker通过轻量级虚拟化技术实现环境隔离,为Flink提供一致的运行环境。本文以Flink 1.20为例,详细讲解如何在单机环境下通过Docker部署完整集群,包括镜像拉取、网络配置、资源调优等关键步骤,并演示经典的WordCount示例。针对开发者常见的内存分配、端口冲突等问题提供解决方案,帮助快速搭建可用于本地开发或测试的Flink环境。

aofan9566的博客 502

windows10环境下nc命令工具

windows环境下nc命令工具,netcat 1.11 for Win32/Win64

netcat-win32-1.11_hellowword_

How to print "Hello word" using C++

netcat-win32-1.12.7z

网络测试工具

【云迁移工具RDA】迁移常见问题

Agent无法正常启动,Linux系统提示“SMS.0201 Failed to start sms agent!”,Windows系统提示“SMS.0201 源端主机网络繁忙,无法连接api网关”。该问题通常是由于源端没有通公网,或者没有按照要求配置hosts,无法与iam或者sms通信导致。需要排查源端网络。1.登录源端服务器。2.检查源端是否能正常访问互联网,若无法访问,请处理后重新启动Agent。可参考弹性公网IP Ping不通检查源端的DNS,防火墙,安全组、本地网络等。

2404_89358774的博客 1486

ros2 gazebo仿真阿克曼小车搭载livox mid360雷达流程

流程简介:仿真环境的搭建需要安装ROS2和Gazebo。使用的是Humble版本,需要确认是否已经正确安装。然后是阿克曼小车模型的URDF,需要包括底盘、转向机构、驱动轮等,需要参考现有的阿克曼模型,比如ClearPath的Husky,但需要调整参数。Livox Mid360雷达在Gazebo中的仿真可能没有现成的插件,需要使用通用3D LiDAR插件来模拟。IMU的仿真相对简单,但需要注意噪声模型,以接近真实数据。

weixin_43802726的博客 3211

NCCL使用指南-进阶篇

前文介绍了NCCL的常用环境变量,本文介绍了NCCL-test性能基准测试,以及一些大规模集群测试的经验数据。

father_yingying的博客 1406

【深度解析】信息素养提升实践活动(附:历年获奖作品分析)

全国师生信息素养提升实践活动是由教育部教育技术与资源发展中心(中央电化教育馆)主办,面向全国师生的一项信息化交流展示活动。

m0_74417957的博客 2628

绿联UGOS Pro系统虚拟机部署Windows 11详细教程(跳账户登录,安装内存驱动)

总的来说,此次的虚拟机部署Windows 11出现的问题应该是目前几乎所有虚拟机都可能出现的问题,和绿联是没有关系的。所以说如果你在其它NAS上虚拟机部署Windows 11系统如果出现同样的问题,那么今天教程的方案也同样适用于你。

qq_63499861的博客 1万+

华为OD机试真题 新系统 2026-08-02 C++ 实现【战士技能规划设计】

你为一款动作游戏设计战士角色的技能。战士每个技能会消耗不同能量,释放技能有两个约束:连续释放技能数量不能超过 m 个,技能能量总和不能超过能量上限 k;如果超过则必须中断当前技能,进入调息状态,也就是分段。

MISAYAONE的博客 75

把事务数据变成可分析数据:用 ABAP CDS 构建 Cube View 与 Query View,并拆解 OData 取数的底层路径

ABAP CDS分析建模:Cube与Query视图的实践解析 本文探讨了ABAP CDS分析建模中Cube View与Query View的核心区别与应用场景: 架构定位 Cube View作为分析型数据源,定义维度和度量的基础语义 Query View作为消费投影,控制查询可见字段和聚合逻辑 关键实现 通过@Analytics.dataCategory: #CUBE声明分析数据源 使用@DefaultAggregation确保指标口径一致性 @Analytics.query: true启用分析引擎特性 运

2007 年 ~ 2025 年,深耕 SAP 技术 18 年 141

【数字IC验证进阶】正则表达式(Regular Expression)快速入门指南

在芯片开发过程中,正则表达式的使用非常常见。初次上手晦涩难懂,多用几次爱不释手!

悟已往之不谏 知来者之可追 1440

TMS_WebGMaps_1.9.8.0_Full_Source

TMS_WebGMaps_1.9.8.0_Full_Source 全源码 delphi cbuilder

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

Statistics and Machine Learning Toolbox

Statistics and Machine Learning Toolbox(带书签),MATLAB统计学习工具

Outlook Express 6 (OE6) for Win7/Win8/Win10

英文版 Outlook Express6.0, 兼容Win7/Win10/WinXP,解压就可用。

上一篇: flink系列1 程序win上启动
下一篇: import org.apache.flink.streaming.api.scala._
dataastron
博客等级 码龄10年 26粉丝 169原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值