python超级计算机_ClusterPool-允许在超级计算机集群上使用calculate（）函数映射大型对象列表-Joseph Goodknight...

阿卡好方法

于 2021-02-11 03:44:47 发布

阅读量188

点赞数

CC 4.0 BY-SA版权

文章标签： python超级计算机

本文链接：https://blog.youkuaiyun.com/weixin_42476912/article/details/113990947

ClusterPool是一个用于SLURM超级计算机调度器的并行计算工具，也可用于普通计算机测试。它提供了一个map函数，能够处理计算任务的分配、持久化存储和收集已完成的任务。Trawler对象负责选取并计算对象，通过文件系统信号同步工作，目前仅支持调用名为`calculate`且无参数的函数。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

作者:Joseph Goodknight

作者邮箱:joey.goodknight@gmail.com

首页:https://github.com/jgoodknight/ClusterPool/

文档:None

下载链接 UNKNOWN

ClusterPool

Provides a map function which works with the SLURM supercomputer scheduler and can be extended to others. Also works on "normal" computers, mainly for testing purposes. (see examples/testfile_primes.py)

A ClusterPool object is an object which gives you a map function. You tell what kind of architecture the ClusterPool exists on (currently only "SLURM" and "normal") and it creates a dispatcher object which keeps track of the objects which need calculating, saves them to disk and collects them when they are done being calculated.

These objects are calculated by "Trawler" objects which get spun to life by the dispatcher and are given a directory full of pickled python objects. The Trawlers then randomly pick an object and throw up a sempahore on the filesystem to tell the other Trawlers they are working on that object. They load the object and the run object.calculate() (Note that currently the ClusterPool package can ONLY handle calling functions called calculate which take no arguments)

Copy from pypi.org