Elasticsearch 8.5 Query with Scripted Metric Aggregation Returns Null for Some Documents
I'm not sure how to approach I'm experiencing an scenario with a scripted metric aggregation in Elasticsearch 8.5 where it intermittently returns `null` for some documents. The goal is to calculate a custom metric based on a combination of fields, but it seems that for certain documents, the script doesn't execute as expected. Here’s the aggregation query I'm using: ```json { "size": 0, "aggs": { "custom_metric": { "scripted_metric": { "init_script": "state.values = []", "map_script": "if (doc['field1'].size() > 0 && doc['field2'].size() > 0) { state.values.add(doc['field1'].value + doc['field2'].value) } else { state.values.add(null) }", "combine_script": "return state.values", "reduce_script": "double total = 0; for (s in states) { for (val in s) { if (val != null) total += val } } return total" } } } } ``` I've verified that `field1` and `field2` exist in the mappings and are populated for the majority of documents. However, I noticed that the `map_script` sometimes doesn't add values to `state.values`. The logs indicate that no errors were thrown, yet the result contains `null` values. To debug, I added logging within the script, but it seems that the documents causing the scenario are simply not entering the `if` condition. I checked the document structure and confirmed that some documents do not have `field1` or `field2`, but I expected the script to handle those cases and return `null` instead of skipping them entirely. I’ve tried modifying the `if` condition to check specifically for `null` values: ```java if (doc['field1'].size() > 0 && doc['field2'].size() > 0 && doc['field1'].value != null && doc['field2'].value != null) { ... } ``` However, this hasn’t solved the scenario. I would appreciate any insights on how to ensure that the aggregation returns a meaningful result while handling documents with missing fields effectively. Is there a better approach to handling missing values in scripted aggregations? I'm on Ubuntu 22.04 using the latest version of Json.