[Security Solution][Detections] Alerts mappings conflict with experimental ECS threat fields

[rylnd]

Summary

As CTI ECS fields and CTI features within SecSol continue to develop in parallel, changes during the RFC process are inevitably going to result in work on the SecSol side. elastic/ecs#1386 is the first example of such work!

While the history here is somewhat complicated, the summary is that:

1. our threat.indicator enrichment data needs to move to a new field (currently: threat.enrichments, as proposed by the stage 1 RFC)
2. our current threat.indicator mapping will now be in conflict with the "official" ECS mappings, as threat.indicator is not a nested field

Action Required

1. We will need to migrate our existing enriched alerts' threat.indicator fields to the new location with an update_by_query or something similar
   • If this functionality is not going to be provided by RAC, we would need to develop data migration as part of the existing migration API
2. Regarding the conflict above in 2., I think we have the following options:
   1. Do nothing, leave the mapping as it is with the ECS conflict
      • This would prevent one from searching threat.indicator across both our alerts indices and other ECS-compliant indices containing the "official" mapping
      • NB also that the threat.indicator mappings are still in stage 2 and are not yet official)
      • We could note this limitation and make the corresponding data migration as part of the RAC alerts migration
   2. Rollover the alerts index to the new, correct mappings
      • This will have the same effect as above, except the conflict would only be encountered when searching across the older indices
      • Users could then leverage the existing migration API to reindex their older alerts data and remove the conflicting mapping
   3. Rename the ECS threat.indicator field to something else prior to the ECS release
      • This would allow us to sidestep the conflict above, as our threat.indicator mapping would be both unused and not in conflict with official ECS.

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[rylnd]

As the indicator and enrichment RFCs continue to make progress, we're going to be moving this issue to 7.15, with the expectation that those RFCs will be official as part of ECS 1.11.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)