Deadletter indexing should be taken care of automatically

[philippkahr]

Hi,

this issue is sparked by a discussion in Slack. Filebeat supports a deadletter output to index any events that ES fails todo so.

As of now one can implement the following workaround, discovered by @herrBez .

1. Go to output settings for Elasticsearch
2. Add the following pieces:

 non_indexable_policy.dead_letter_index:
    index: "logs-unindexable-default"

3. You need to create an index template for logs-unindexable-* where you disable any mapping.
4. Now since the Agent isn't actually allowed to write to logs-unindexable-default because it does not have an API key.
5. You need to create a custom log integration and put the unindexable as the dataset name, so it will allow the agent to write to that index.

---

I think that is not a good UX, not very straightforward and very error prone. I would hope that the shipper / agent is responsible directly to enable this feature and with a single toggle out of Kibana I can enable a logs-unindexable, metrics-unindexable, automatically and no need for manual intervention.