Optimizing large batch updates in Django ORM - experiencing slow performance
Does anyone know how to I've encountered a strange issue with I've been struggling with this for a few days now and could really use some help... I'm currently working with performance optimization when trying to perform large batch updates using Django's ORM. I'm using Django 3.2 and PostgreSQL 12, and I've noticed that updating around 10,000 records at once is taking much longer than expected. I tried using `bulk_update`, but my updates are not working as intended because they seem to skip some fields if they are not included in the list of objects I'm passing. Here’s a snippet of what I’ve been trying: ```python from myapp.models import MyModel from django.db import transaction # Assume we have a list of updated values for each object objects_to_update = [] for obj in my_queryset: obj.field1 = new_value1 obj.field2 = new_value2 objects_to_update.append(obj) with transaction.atomic(): MyModel.objects.bulk_update(objects_to_update, ['field1', 'field2']) ``` The question is that even though this is within a transaction, the performance is still poor. I also tried using raw SQL with `execute`, but I’m not sure if that’s the best practice and it feels less safe. Additionally, I ran into a `UniqueViolationError` when I attempted to execute an update that inadvertently created duplicate key scenarios. Are there any recommended strategies or optimizations for efficiently updating a large number of records in Django without compromising data integrity? Should I consider alternative approaches or perhaps even using Django signals to optimize this operation? I’ve read that using the `update()` method could be faster, but I’m unsure how to structure it properly for multiple records. I’d appreciate any guidance! My development environment is Ubuntu. Is there a better approach? For context: I'm using Python on Linux. The project is a mobile app built with Python. Any examples would be super helpful. I recently upgraded to Python 3.10. Am I missing something obvious?