flink启动(updating)

最新推荐文章于 2024-03-28 17:04:37 发布

大怀特

最新推荐文章于 2024-03-28 17:04:37 发布

阅读量216

点赞数

CC 4.0 BY-SA版权

分类专栏： stream 文章标签： flink

本文链接：https://blog.youkuaiyun.com/rustwei/article/details/120459445

stream 专栏收录该内容

17 篇文章

订阅专栏

本文介绍如何在Flink中设置并行度和slot数，通过调整这些参数来优化任务执行效率。详细解释了taskmanager和jobmanager所需资源的计算方式，并提供了具体的命令示例。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

设置taskmanager并行度及slot

task_manager数量 = 并行度数量(parallelism.default) / slot数量(slots) 
jobmanager所占资源为额外1个

bin/yarn-session.sh -d -Dparallelism.default=48 --name app_wsw --slots 6 --taskManagerMemory 6114 --jobmanager yarn-cluster --jobManagerMemory 4096

Running Containers	Allocated CPU VCores
9	49

flink <ACTION> [OPTIONS] [ARGUMENTS]

bin/flink run -t yarn-session -ynm wsw -yjm 2048MB -ytm 1024MB -ys 6
    
bin/flink run-application -t yarn-session -ynm application_wsw -yjm 2048MB -ytm 4096MB -ys 6
    
$FLINK_HOME/bin/flink run -t yarn-per-job \
    -Dyarn.application.name="rec_No1PharmacyAssociatedWordGoodsStandardName_wsw" \
    -yjm 2048MB -ytm 4096MB -ys 6 \
    -c com.bl.bigdata.recommedation.supportflink.search.No1PharmacyAssociatedWordGoodsStandardName \
    /home/bigdata_tec/sven/recommedation_support_flink-1.0-SNAPSHOT.jar

    bin/flink run-application -t yarn-application \
    -ynm application_wsw -yjm 2048MB \
    -ytm 4096MB -ys 8 \
    -c com.bl.bigdata.recommedation.supportflink.search.No1PharmacyAssociatedWordGoodsStandardName \
    /home/bigdata_tec/sven/recommedation_support_flink-1.0-SNAPSHOT.jar

$FLINK_HOME/bin/flink run-application -t yarn-application \
    -c com.bl.bigdata.recommedation.supportflink.search.No1PharmacyAssociatedWordGoodsStandardName \
    -Djobmanager.memory.process.size=2048m \
    -Dtaskmanager.memory.process.size=2048m \
    -Dtaskmanager.numberOfTaskSlots=8 \
    -Dparallelism.default=1 \
    -Dyarn.application.name="rec_No1PharmacyAssociatedWordGoodsStandardName_wsw" \
    /home/bigdata_tec/sven/recommedation_support_flink-1.0-SNAPSHOT.jar

$FLINK_HOME/bin/flink run -t yarn-per-job -Dyarn.application.name="search_associatedWord_wsw" -d \
	-Dtaskmanager.memory.process.size=16384MB \
    -Djobmanager.memory.process.size=4096MB -Dtaskmanager.numberOfTaskSlots=2 \
    -Dparallelism.default=4 \
    -Ddfs.client.block.write.locateFollowingBlock.retries=10 \
    -c com.bl.bigdata.recommedation.supportflink.search.FastGoHomeGoodsAssociatedWord  \
    -s  hdfs:///tmp/flink/checkpoints/11049ff36bef4e7673cda5e1ac5b0d0c/chk-243852 /var/lib/hive/sven/recommedation_support_flink-1.0-SNAPSHOT.jar
     
$FLINK_HOME/bin/flink run -t yarn-per-job \
-Dyarn.application.name="cdp_clientGroup_wsw" \
-Dtaskmanager.memory.process.size=16384MB \
-Djobmanager.memory.process.size=4096MB -Dtaskmanager.numberOfTaskSlots=1 \
-Ddfs.client.block.write.locateFollowingBlock.retries=10 \
-d \
-c com.bl.bigdata.cdp.cdpflink.CDPClientGroupSimple  \
-s hdfs:///tmp/flink/cdp/client_group/checkpoints/1f576c7409fc1b709b4e5e7eda87e1e8/chk-5205 \
/var/lib/hive/sven/cdpflink-0.1.jar

flink启动资源分配

设置taskmanager并行度及slot

task_manager数量 = 并行度数量(parallelism.default) / slot数量(slots)
jobmanager所占资源为额外1个

bin/yarn-session.sh -d -Dparallelism.default=48 --name app_wsw --slots 6 --taskManagerMemory 6114 --jobmanager yarn-cluster --jobManagerMemory 4096

在这里插入图片描述