celery初探

我们用的是gearman，但对于类似功能的celery也要做一下调研。

Celery

celery：celery是一个分布式的任务调度模块，可以支持多台不同的计算机执行不同的任务或者相同的任务。

安装

版本要求：

Version Requirements

Celery version 5.0 runs on

Python ❨3.6, 3.7, 3.8❩

PyPy3.6 ❨7.3❩

Celery 4.x was the last version to support Python 2.7, Celery 5.x requires Python 3.6 or newer.


If you’re running an older version of Python, you need to be running an older version of Celery:

Python 2.7 or Python 3.5: Celery series 4.4 or earlier.

Python 2.6: Celery series 3.1 or earlier.

Python 2.5: Celery series 3.0 or earlier.

Python 2.4 was Celery series 2.2 or earlier.

Celery is a project with minimal funding, so we don’t support Microsoft Windows. Please don’t open any issues related to that platform.

由于我们的python版本有3.5和3.6，起初，为稳妥起见，还是安装了Celery4.4。

但实测下来，发现linux下py3.5对Celery4.4的支持依然不好，出各种问题，所以，最终还是切到py3.6，Celery版本使用最新的5.0.5

hello world

celery worker端代码：

import time
from celery import Celery

app = Celery('tasks', broker='redis://xx.xx.xx.xx:6379/0')

@app.task
def sendmail(mail):
    print('sending mail to %s...' % mail['to'])
    time.sleep(2.0)
    print('mail sent.')

windows上执行直接报错：not enough values to unpack (expected 3, got 0)，google之，发现是celery4以后对windows的支持不好。改到linux上执行就没问题了。

注意：启动celery worker会提醒你安装redis库，因我们使用redis来作为消息队列。用

pip3 install redis

安装好即可。

celery client端代码：

from celery_task import sendmail
sendmail.delay(dict(to='celery@python.org'))

执行完成，worker端打印日志：

[tasks]
  . celery_task.sendmail

[16:14:03,227: INFO/MainProcess] Connected to redis://xx.xx.xx.xx:6379/0
[16:14:03,256: INFO/MainProcess] mingle: searching for neighbors
[16:14:04,306: INFO/MainProcess] mingle: all alone
[16:14:04,364: INFO/MainProcess] celery@localhost.localdomain ready.
[16:15:02,842: INFO/MainProcess] Received task: celery_task.sendmail[940ddec2-6a74-437c-9f4c-26c07849cd8f]  
[16:15:02,846: WARNING/ForkPoolWorker-128] sending mail to celery@python.org...
[16:15:04,848: WARNING/ForkPoolWorker-128] mail sent.

从上述例子看出，celery跟gearman还是很像的，只不过，celery写起来更像RPC，而gearman使用起来更像通信协议。gearman有自己的server做中转，供client提交任务、供worker抓取任务，celery则依赖三方队列（如redis或mq）中转。gearman在client–server、server–worker间使用自己的通信协议，celery则使用mq的API。