👀 Views: 58 💬 Answers: 1 📅 Created: 2025-06-27

R data.table performance

I'm deploying to production and I'm experimenting with I'm following best practices but I'm working with a performance scenario while using `lapply()` to apply a function to a large list of data frames that I have stored in a `data.table` format..... My goal is to perform a transformation on each data frame in the list, but the process is significantly slower than I expected, particularly with larger data frames. I've tried using `data.table` for its speed advantages but still see sluggish performance. Here’s a small snippet of the code I’m using: ```r library(data.table) # Sample list of data.tables my_list <- lapply(1:100, function(x) data.table(a = rnorm(1000), b = rnorm(1000))) # Function to transform each data.table transform_function <- function(dt) { dt[, c := a + b] return(dt) } # Applying the function using lapply result <- lapply(my_list, transform_function) ``` While this works, it takes quite a long time to complete. I tried switching to `lapply` with `future.apply` to parallelize the operation: ```r library(future.apply) plan(multisession) # Using future_lapply to parallelize result <- future_lapply(my_list, transform_function) ``` However, I still experience performance bottlenecks. I’ve also checked that my R version is 4.3.0, and I'm using `data.table` version 1.14.4. Are there any optimizations or alternative approaches to speed up this type of operation on a large list of data frames? Any insights on best practices for handling such cases would be greatly appreciated! The project is a REST API built with R. Any feedback is welcome! I'm working on a web app that needs to handle this.