Django File Uploads - Handling Large Files with Streaming and Progress Tracking
I'm optimizing some code but I've hit a wall trying to Hey everyone, I'm running into an issue that's driving me crazy... I'm working on a Django application that allows users to upload large files (up to 2GB)... I've implemented a file upload view using Django's built-in file handling, but I'm running into performance optimization and timeouts when users attempt to upload these files. Currently, the view looks something like this: ```python from django.http import JsonResponse from django.views import View from django.core.files.storage import FileSystemStorage class FileUploadView(View): def post(self, request): uploaded_file = request.FILES['file'] fs = FileSystemStorage() filename = fs.save(uploaded_file.name, uploaded_file) return JsonResponse({'file_name': filename}) ``` When a user tries to upload a large file, the request often times out, and I see this behavior in the logs: ``` [behavior] The operation timed out after 30 seconds. ``` I have also set `DATA_UPLOAD_MAX_MEMORY_SIZE` to `104857600` (100MB) in my settings to allow larger file uploads, but this still doesn’t seem to help with the timeout scenario. To mitigate this, I attempted to implement file streaming using Django's `StreamingHttpResponse`, but I found it challenging to track the upload progress. My aim is to provide users with feedback on how much of the file has been uploaded. I would like to know the best practices for handling large file uploads in Django, specifically regarding streaming and tracking upload progress. Here's what I've considered so far: 1. Using Django REST Framework's `FileUploadParser` for better handling. 2. Implementing a frontend solution using JavaScript to show progress, but I need a reliable way to know the upload progress on the server side. Can anyone provide guidance on how to effectively manage large file uploads in Django, ensure the server can handle the requests without timing out, and give accurate feedback to users during the upload process? I'd really appreciate any guidance on this. Thanks, I really appreciate it! This is happening in both development and production on Debian. Thanks in advance! What would be the recommended way to handle this?