Elasticsearch 8.5 Scripted Metric Aggregation Unexpected Results with Large Datasets
I'm trying to configure I'm updating my dependencies and I'm having trouble with I'm stuck on something that should probably be simple. I'm working through a tutorial and I'm using Elasticsearch 8.5 and trying to implement a scripted metric aggregation to calculate the average duration of events stored in a large dataset... However, I'm working with unexpected results when the dataset exceeds a certain size. For datasets with less than 10,000 documents, the calculation works as intended, but with larger datasets, the returned average seems to be skewed, often higher than expected. Here's my aggregation query: ```json { "size": 0, "aggs": { "average_duration": { "scripted_metric": { "init_script": "state.durations = []", "map_script": "if (doc['duration'].size() > 0) { state.durations.add(doc['duration'].value) }", "combine_script": "return state.durations", "reduce_script": "double sum = 0; int count = 0; for (s in states) { for (duration in s) { sum += duration; count++; } } return sum / count;" } } } } ``` I've tried to debug the scenario by logging the `state.durations` array at each stage, and it seems to collect the correct values during the map phase, but during the reduce phase, the counts seem off, especially when I increase the size of my index. Additionally, I'm working with a timeout scenario when the dataset becomes large, which causes the aggregation to unexpected result. I attempted to adjust the `timeout` parameter as follows, but the question continues: ```json "timeout": "30s" ``` I've also ensured that my cluster has sufficient resources allocated, but it appears that as more documents are added, the performance degrades significantly. What could be causing this scenario with the scripted metric aggregation, and how can I optimize it to handle larger datasets effectively? My development environment is Windows. I'm open to any suggestions. My development environment is Debian. My development environment is Windows 11. Thanks for any help you can provide! How would you solve this?