Celery异步任务处理-优快云博客

本文链接：https://blog.youkuaiyun.com/u012279631/article/details/83473213

celery

celery 是python 做异步通信的工具包。首先我们要明白异步的原因，那就是用户可以允许的情况下，把耗时的操作拆分出去，这样子做到了当前的快速响应。同时也避免了用户一个耗时的操作因为内部资源的不足而出现超时的情况。celery使用了非常简单的方式实现了异步。主要是celery sender broker 到consumer的过程。

安装celery

pip install celery

不用怀疑以上就是安装celery的过程

celery 使用

首先需要创建app

from celery import Celery
app = Celery('tasks', broker="amqp://", backend='amqp')

导入celery 包，使用Celery 类，实例化celery类。参数上第一个是名字，broker指定使用的异步队列组件是哪个，backend是结果返回到哪里去。以上是使用本地的rabbitmq作为消息队列的组件。刚网上的例子也是rabbitmq。

这时候我们就用实例化了一个Celery。

实现一个异步任务

@app.task
def add(x, y):
	print 'into function ...'

	time.sleep(4)
	return x + y

注意，在这个方法之上使用了@app.task 装饰器，这也是我们要实例化app的理由之一。使用了该装饰器可以使其成为异步task。

发送异步任务

from task import add


if __name__ == '__main__':
	print 'start task...'
	for i in range(100):
		result = add.delay(2, 4)
	print 'end task..'
	print result

这里需要注意的是delay方法，使用这个方法才能够使用异步的方式调用add方法，否则还是同步调用。

启动celery 消费者

在这里如果我们直接run主程序，返回结果是这样的

start task...
end task..
cc44a987-a875-4271-8fc3-c5cd8b068c18

下面那一串相当于是requestID, 他已经丢到了rabbitmq上。可以通过以下方式查看本地的rabbitmq。

http://localhost:15672/#/queues

结果如下图。

这时候看到我们执行了两次之后相当于有200个队列在等待消费。

启动消费者需要如下命令：

celery worker -A task -l info

启动后输出

guifeng.chen  /Users/guifeng.chen/program/testdj/celery_task   celery worker -A task -l info                                                              22:56:10

/Library/Python/2.7/site-packages/celery/backends/amqp.py:67: CPendingDeprecationWarning:
    The AMQP result backend is scheduled for deprecation in     version 4.0 and removal in version v5.0.     Please use RPC backend or a persistent backend.

  alternative='Please use RPC backend or a persistent backend.')

celery@GuifengdeMacBook-Pro.local v4.2.1 (windowlicker)

Darwin-16.7.0-x86_64-i386-64bit 2018-09-24 22:56:22

[config]
.> app:         tasks:0x1101165d0
.> transport:   amqp://guest:**@localhost:5672//
.> results:     amqp://
.> concurrency: 8 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.> celery           exchange=celery(direct) key=celery


[tasks]
  . task.add

[2018-09-24 22:56:23,817: INFO/MainProcess] Connected to amqp://guest:**@127.0.0.1:5672//
[2018-09-24 22:56:23,832: INFO/MainProcess] mingle: searching for neighbors
[2018-09-24 22:56:24,866: INFO/MainProcess] mingle: all alone
[2018-09-24 22:56:24,880: INFO/MainProcess] celery@GuifengdeMacBook-Pro.local ready.
[2018-09-24 22:56:25,139: INFO/MainProcess] Events of group {task} enabled by remote.

可以看到config里面的一些配置 concurrency 是并发的线程数量。
queues 队列是rabbitmq上的队列名称。

[2018-09-24 22:58:35,016: INFO/MainProcess] Received task: task.add[4d7ab8e2-7f13-4481-974d-bbe0574424ad]
[2018-09-24 22:58:35,017: INFO/MainProcess] Received task: task.add[364f47d5-d032-4194-9208-36f1970ad34b]
[2018-09-24 22:58:35,018: INFO/MainProcess] Received task: task.add[ed0eda28-5237-4ba5-ab6f-7fa4e9bf1d55]
[2018-09-24 22:58:35,019: INFO/MainProcess] Received task: task.add[725694d5-6d10-4905-b854-af108cedd8a9]
[2018-09-24 22:58:35,019: INFO/MainProcess] Received task: task.add[62fc4629-9f35-41b4-8b2d-7cf882c69a45]
[2018-09-24 22:58:35,020: INFO/MainProcess] Received task: task.add[5013ef95-6d8f-4312-9ce7-5428a9440721]
[2018-09-24 22:58:35,020: WARNING/ForkPoolWorker-1] into function ...
[2018-09-24 22:58:35,020: WARNING/ForkPoolWorker-2] into function ...

输出类似这种打印表明已经在消费了。

启动目录

这里需要注意的是启动celery的目录有一定的限制

 celery worker -A celery_task.task -l info

比如从更上级的目录启动了celery。这个时候他的路径会不一样。

[config]
.> app:         tasks:0x1024915d0
.> transport:   amqp://guest:**@localhost:5672//
.> results:     amqp://
.> concurrency: 8 (prefork)
.> task events: OFF (enable -E to monitor tasks in this worker)

[queues]
.> celery           exchange=celery(direct) key=celery


[tasks]
  . celery_task.task.add

可以看到他的task路径就变成了 celery_task.task.add

[2018-09-25 09:32:18,414: INFO/MainProcess] celery@GuifengdeMacBook-Pro.local ready.
[2018-09-25 09:32:18,414: ERROR/MainProcess] Received unregistered task of type 'task.add'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you're using relative imports?

Please see
http://docs.celeryq.org/en/latest/internals/protocol.html
for more information.

The full contents of the message body was:
u'[[2, 4], {}, {"chord": null, "callbacks": null, "errbacks": null, "chain": null}]' (81b)
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/celery/worker/consumer/consumer.py", line 558, in on_task_received
    strategy = strategies[type_]
KeyError: 'task.add'

消费的时候就会报错,因为注册的是task.add。