Celery: keep a task queue from becoming a black box

Marketing launches email campaign to 50,000 contacts. Interface says "scheduled send." Three hours later: 200 emails sent. Flower unreachable, Redis emails queue shows 48,000 waiting messages, and PDF export workers monopolize pool since morning cron. Nobody alerted on queue depth — Celery "worked," jobs waited in the dark.

Celery decouples async — it moves visibility. Without broker monitoring, retry policy and queue separation, you replaced slow HTTP request with invisible backlog that rots business hours later.

Observable architecture

Minimum alerts:

• queue_length > 1000 for ten minutes
• Task failure rate spike
• Missing worker heartbeat

Task without max timeout occupies worker until reboot.

Healthy task configuration

@app.task(bind=True, max_retries=3, default_retry_delay=60,

          soft_time_limit=300, time_limit=360)

def send_campaign(self, batch_id):

    ...

• acks_late=True: ack after success — requeue if worker crashes mid-task.
• reject_on_worker_lost depending on broker.
• Routing: heavy jobs to heavy queue, emails to fast.

Prefetch: worker_prefetch_multiplier=1 for long jobs — prevents worker hoarding queue.

Dead letter and failure inspection

After max retries:

• Store exception and args in database (django-celery-results).
• Notify the operations team on a dedicated channel with enough context to restart cleanly.
• Provide admin UI for manual requeue after fix.

Do not retry forever — poison message blocks slot indefinitely.

Broker and hosting

Redis broker: simple, often on dedicated Redis VPS; AOF persistence if messages must survive reboot.

RabbitMQ: advanced routing, more operational load.

Celery workers run as separate processes from Gunicorn — often same VPS for small project, second machine if load grows. Few managed hosts offer native Celery: you manage workers on VPS or cloud.

For scheduled tasks feeding Celery, cross-check reliable cron and Gunicorn nginx.

The peak: Celery without dashboard is lying cron

Here is what "task went through Celery" hides in meetings.

Definition of done for a Celery task: queue depth metric, alert, and documented requeue procedure — not just the @shared_task decorator.

Decide and move forward without blind spots

In one day, you can make Celery observable without rebuilding the entire architecture:

1. Deploy Flower or a Prometheus exporter in production, not only on the development environment where nobody watches queues at night.
2. Separate queues by criticality and duration — transactional email, heavy exports, overnight maintenance — so a long job does not block an urgent send.
3. Apply execution timeouts and a maximum retry count to all existing tasks, including those written two years ago.
4. Configure an alert when queue depth exceeds a threshold for ten to fifteen consecutive minutes.
5. Test a brutal worker stop mid-task and verify the message returns to the queue.
6. Document who restarts a blocked queue, under what timeline, and with what verification procedure after restart.

Compare VPS suitable for workers via the directory and comparison tool.

Frequently asked questions

How to know if Celery tasks are piling up?

Monitor broker queue length with the right commands (for example LLEN on Redis or the RabbitMQ management UI), the Flower dashboard, and reserved or active task metrics. Set an alert if depth stays high for more than fifteen minutes. The site can return 200 OK while thousands of jobs wait without the business team noticing.

What to do with repeatedly failing tasks?

Set up a dead letter queue, enable task_acks_late with a maximum retry count and progressive backoff, then store failures in an inspectable database such as django-celery-results. Never retry forever: a poison message monopolizes a worker. Fix the root cause before any mass requeue from the admin interface.

Redis or RabbitMQ as broker?

Redis suits SMBs and small projects thanks to simple setup. RabbitMQ wins when routing is complex, message persistence is critical, or several consumers share queues with fine-grained rules. In both cases, monitor broker server memory and message persistence after reboot.

How many Celery workers?

Size dedicated pools per queue type, for example -Q emails,exports, rather than a single pool. Heavy exports must not share the same pool as transactional email — otherwise an urgent send can wait two hours behind a ten-minute PDF job.

Transparent Celery shows the queue — black box Celery fills it while site returns 200 OK.