Description
This is a feature request.
I'm migrating a platform from Apache Solr to ElasticSearch and I'm struggling in creating dynamic fields like Apache Solr used to have. I need facets (aka "aggregations" in ElasticSearch) on these dynamic fields. On Solr that was easily done with an annotation on a Map:
@Dynamic("*_str")
@Field("attributes")
private Map<String, String> attributes;Instead, with ElasticSearch, I need to create a dynamic template for the index like this:
{
"mappings": {
"dynamic_templates": [
{
"_str": {
"match": "*_str",
"match_mapping_type": "string",
"mapping": {
"type": "keyword"
}
}
}
]
}
}The main issue is that when I now save a document having
@Field("attributes")
private Map<String, String> attributes;, Spring is inputting a payload like
{
// other document fields above
"attributes": {
"code1_str":"value1",
"code2_str":"other-value"
},
// other document fields below
}and I can't do any aggregation query to get a result like
{
"aggregations": {
"code1_str": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "value1",
"doc_count": 1
},
{
"key": "value2",
"doc_count": 1
}
]
},
"code2_str": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "other-value",
"doc_count": 2
}
]
}
}
}The workaround for this issue is atm a custom Spring Converter<S, T> for the entire document: immagine having to convert 20 or 30 fields.. this would take a lot of work just because of 1 field serialization/deserialization issue.
A solution could be found Instead in having something like @JsonUnwrapped annotation from Jackson library to remove the attributes level from the payload, leading to Spring inputting a payload like:
{
// other document fields above
"code1_str":"value1",
"code2_str":"other-value"
// other document fields below
}Could this be feasible in an upcoming release?
Metadata
Metadata
Assignees
Labels
Type
Projects
Milestone
Relationships
Participants
Activity
sothawo commentedon Nov 26, 2023
When pulling out the properties from a map into the entity that contains the map, how should this work when reading entities from Elasticsearch?
This works when writing data, but on reading it is not possible to determine which of the fields returned from Elasticsearch should go into properties and which into the map, there is no general way to determine where
code1_strshould be put into. What if the entity has two maps as properties? In which one should this be read?What would happen on writing, when an entity has a property
prop1, and the map that should be unwrapped as well has a keyprop1? Which one should be written and which one overwritten?AlcipPopa commentedon Nov 29, 2023
Hmm yes you're right: the issue is on serialization from JSON to POJO.
Would a wildcard ease the mapping of fields between POJOs and ElasticSearch documents? This would also require a validation of the POJO field names before saving data to ElasticSearch, to avoid having multiple matching fields.
So this should be possible to save on ElasticSearch:
{ // other document fields above "code1_str":"value1", "code2_str":"other-value" // other document fields below }While this should throw a RuntimeException before even saving into ElasticSearch, because of ambiguous mapping:
P.S.: Don't mind the @dynamic annotation since it's Solr related.
sothawo commentedon Dec 1, 2023
We'd need to use some new annotation to prevent a mappping to be written for these fields. That idea probably will work.
I'm not yet sure if this can be done only for primitive types in the map like
Map<String, String>or if this can work for something likeMap<String,E>, might be possible.AlcipPopa commentedon Dec 4, 2023
I wouldn't use a generic value for the map.. at least not for a 1st version :)
youssef3wi commentedon Aug 16, 2024
You can use the
@DynamicTemplatesannotation can generate the mapping definition you need.After that the converter should handle the de/serialization process between the Elasticsearch request/response and the Document (Java object) to manage this definition.