Missing fields for terms aggregations

[peacand]

Hi,

I'm trying to perform some terms aggregations on multiple fields against documents having different formats and fields.
For example I would like to run terms aggregations on fields "user" and then "event_type".
If the documents matching my query don't have a field "user", then the result of the whole aggregation is empty, even if my documents have a field "event_type".

I would like to be able to configure a default value for non existing fields.
For example, if I set the default value for missing fields to "NULL", the result of such aggregation would be :

| user | event_type | docs_count |
| NULL | value1 | 789789
| NULL | value2 | 678678

In case part of the documents has both fields and another part only has the field "event_type", then the result would be :

| user | event_type | docs_count |
| NULL | value1 | 456
| NULL | value2 | 6778
| user1 | value1 | 78675
| user2 | value1 | 45645

Note: All the docs_count values are completely random, there is absolutely no logic in these values used as examples.

This feature seems to be equivalent to the "missing" parameter in the Elasticsearch aggregation API : https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_missing_value_5

[PSeitz]

Thanks for the bug report.

The missing parameter is unsupported currently, I think we can add this in the next release.

Missing and Mixed Types

One issue we could have is that term aggregation may run on fields with mixed types, e.g. two columns: numbers and text. Each column type has its own column index for existence of values. I think a missing parameter should apply over all.

Aggregations over multiple fields are running independently currently, so this is not so easy to handle.
We could just ignore this corner case for now.

To avoid duplicate missing results, we could apply the missing parameter depending on its type.

Related Issues: quickwit-oss/tantivy#1789 quickwit-oss/tantivy#1913

[PSeitz]

This is implemented on main branch and will be released with quickwit 0.7

quickwit-oss/tantivy#2149
quickwit-oss/tantivy#2103
quickwit-oss/tantivy#2150
quickwit-oss/tantivy#2151
quickwit-oss/tantivy#2157