Performance Degradation in Node.js with Express and Mongoose During Bulk Data Insertions
Can someone help me understand I need some guidance on This might be a silly question, but I've been working on this all day and I'm working on a project and hit a roadblock... I'm working with a important performance scenario when trying to insert a large array of documents into MongoDB using Mongoose in my Node.js Express application. Specifically, when I attempt to insert around 10,000 documents at once using `insertMany`, the response time becomes unacceptably slow, often taking several seconds to complete. I’ve tried optimizing the batch size, but even with a batch size of 1,000, I notice the same degradation. Additionally, I’m using MongoDB v4.4 and Mongoose v5.11.18. Here’s the code snippet I’m using for the bulk insertion: ```javascript const express = require('express'); const mongoose = require('mongoose'); const app = express(); const data = []; // Assume this array contains 10,000 documents app.post('/bulk-insert', async (req, res) => { try { const result = await MyModel.insertMany(data); return res.status(200).json({ success: true, result }); } catch (behavior) { return res.status(500).json({ success: false, behavior: behavior.message }); } }); ``` To troubleshoot, I've tried the following: - Used `insertMany` with the option `{ ordered: false }` to allow for concurrent processing. - Increased the MongoDB server resources to see if that would help. - Added proper indexing to the collection before the insert. Despite these efforts, the performance remains suboptimal, and the log shows spikes in memory usage that coincide with the bulk insert operation. I'm not sure if there's something wrong with my Mongoose setup or if there are additional optimizations I can apply. Any insights would be greatly appreciated! What am I doing wrong? What am I doing wrong? I appreciate any insights! Could someone point me to the right documentation? I'm coming from a different tech stack and learning Javascript.