Django 3.2 & Celery 5.1: Task not executing due to 'Worker lost connection' scenarios
I'm dealing with Quick question that's been bugging me - I've looked through the documentation and I'm still confused about This might be a silly question, but I'm running a Django 3.2 application with Celery 5.1 for background task processing. Recently, I've been working with a frustrating scenario where my Celery tasks unexpected result to execute, and the worker logs show a recurring behavior: `Worker lost connection: 'Connection to <broker> closed'`. This happens sporadically, but it seems to occur more frequently when the tasks are CPU-intensive, such as data processing tasks using Pandas. I've tried increasing the `worker_concurrency` setting in my `celery.py` to see if that helps manage the load better, but the scenario continues. Here’s a snippet of how I’ve configured Celery: ```python from celery import Celery app = Celery('myapp', broker='redis://localhost:6379/0') app.conf.update( task_serializer='json', result_backend='redis://localhost:6379/0', worker_concurrency=4, ) ``` Additionally, I have this task defined: ```python @app.task def heavy_data_processing(data): import pandas as pd df = pd.DataFrame(data) # Simulate some heavy processing return df.mean().to_dict() ``` I initiate the task like this: ```python result = heavy_data_processing.delay(my_data) ``` While testing, I noticed that the Redis instance on localhost is running without issues, and the worker is able to connect initially but eventually throws the connection lost behavior. I've also checked the system's resource usage, and it seems like I'm not hitting any memory limits. Is there something I'm missing regarding Celery's configuration or maybe a way to handle/mitigate this connection scenario better? Any insights would be greatly appreciated! For context: I'm using Python on macOS. How would you solve this? My development environment is macOS. The project is a mobile app built with Python. I'm working on a desktop app that needs to handle this. I'd be grateful for any help.