celery使用的时候的坑

Celery异步任务参数与重复执行问题

最新推荐文章于 2024-04-11 15:53:29 发布

转载最新推荐文章于 2024-04-11 15:53:29 发布 · 1.1k 阅读

1 ·

CC 4.0 BY-SA版权

原文链接：http://www.cnblogs.com/lanlingshao/p/10087280.html

文章标签：

#json

本文探讨了Celery中delay及apply_async函数的参数传递限制，特别是实例对象不可传递的问题，并深入分析了当使用Redis作为broker时，由于visibility_timeout设置不当导致的任务重复执行现象。提供了调整配置避免重复执行的解决方案。

一、delay函数或者apply_async函数的传参问题

1、通过delay或者apply_async传参数给异步任务的时候不能传实例，否则会报错raised unexpected: EncodeError(TypeError('<ExtractCashRecord: ExtractCashRecord object> is not JSON serializable，因为我传的参数有一个是ExtractCashRecord这个类的实例。

二、任务重复执行

celery使用redis作为broker的话，apply_async的异步任务如果执行之间是延迟超过1小时的话，celery会重复发布任务，会导致任务重复执行。这个在官方文档中有说明celery文档

if a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact if that happens it will be executed again, and again in a loop.So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use.Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.Periodic tasks won’t be affected by the visibility timeout, as this is a concept separate from ETA/countdown.You can increase this timeout by configuring a transport option with the same name:
app.conf.broker_transport_options = {'visibility_timeout': 43200}*
The value must be an int describing the number of seconds.

这个时候，我们就需要在配置文件里面增加一项，来增大visibility_timeout的时间
BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 43200}
或者
app.conf.broker_transport_options = {'visibility_timeout': 43200}

转载于:https://www.cnblogs.com/lanlingshao/p/10087280.html