Custom rule in Elastic Security throws RuleDataWriteDisabledError error when run

[brsolomon-deloitte]

Kibana version: 8.5.3

Elasticsearch version: 8.5.3

Browser version: Chrome 108.0.5359.124

Original install method (e.g. download page, yum, from source, etc.): ECK

Describe the bug: Custom rule fails to run with Bulk Indexing of signals failed: RuleDataWriteDisabledError: Rule registry writing is disabled due to an error during Rule Data Client initialization.

Steps to reproduce:

1. Create rule as shown below
2. Ingest Suricata alerts with Filebeat suricata module
3. Watch as the Rule execution log fills up with Failed runs with zero useful debugging information

Expected behavior: Rule runs successfully.

Screenshots (if relevant):

Provide logs and/or server output (if relevant):

Kibana logs:

[2023-01-05T15:27:00.022+00:00][INFO ][plugins.securitySolution.ruleExecution] Changing rule status to "running" [siem.queryRule][Suricata alerts][rule id 43afdd40-8c73-11ed-85b3-87db925d75d3][rule uuid a5377b9a-75f9-446d-bc00-2c5262c9190a][exec id 1e3e038c-3308-4af5-a4d8-5afca2950d36][space default]
[2023-01-05T15:27:00.321+00:00][ERROR][plugins.securitySolution.ruleExecution] [-] search_after_bulk_create threw an error RuleDataWriteDisabledError: Rule registry writing is disabled due to an error during Rule Data Client initialization. [siem.queryRule][Suricata alerts][rule id 43afdd40-8c73-11ed-85b3-87db925d75d3][rule uuid a5377b9a-75f9-446d-bc00-2c5262c9190a][exec id 1e3e038c-3308-4af5-a4d8-5afca2950d36][space default]
[2023-01-05T15:27:00.321+00:00][ERROR][plugins.securitySolution.ruleExecution] Changing rule status to "failed". Bulk Indexing of signals failed: RuleDataWriteDisabledError: Rule registry writing is disabled due to an error during Rule Data Client initialization. [siem.queryRule][Suricata alerts][rule id 43afdd40-8c73-11ed-85b3-87db925d75d3][rule uuid a5377b9a-75f9-446d-bc00-2c5262c9190a][exec id 1e3e038c-3308-4af5-a4d8-5afca2950d36][space default]

The rule in question

Fetched with GET kbn:/api/alerting/rules/_find?search_fields=name&search=Suricata*.

{
  "page": 1,
  "total": 1,
  "per_page": 10,
  "data": [
    {
      "id": "43afdd40-8c73-11ed-85b3-87db925d75d3",
      "name": "Suricata alerts",
      "tags": [
        "Suricata"
      ],
      "consumer": "siem",
      "enabled": true,
      "throttle": null,
      "schedule": {
        "interval": "5m"
      },
      "params": {
        "author": [
          "REDACTED"
        ],
        "description": "Generates a detection alert each time a Suricata alert is received. Enabling this rule allows you to immediately begin investigating your Suricata alerts.",
        "ruleId": "a5377b9a-75f9-446d-bc00-2c5262c9190a",
        "falsePositives": [],
        "from": "now-600s",
        "immutable": false,
        "license": "Proprietary",
        "outputIndex": "",
        "meta": {
          "from": "5m",
          "kibana_siem_app_url": "https://REDACTED/app/security"
        },
        "maxSignals": 100,
        "relatedIntegrations": [],
        "requiredFields": [],
        "riskScore": 0,
        "riskScoreMapping": [],
        "ruleNameOverride": "rule.name",
        "setup": "",
        "severity": "medium",
        "severityMapping": [
          {
            "severity": "low",
            "field": "event.severity",
            "value": "4",
            "operator": "equals"
          },
          {
            "severity": "medium",
            "field": "event.severity",
            "value": "3",
            "operator": "equals"
          },
          {
            "severity": "high",
            "field": "event.severity",
            "value": "2",
            "operator": "equals"
          },
          {
            "severity": "critical",
            "field": "event.severity",
            "value": "1",
            "operator": "equals"
          }
        ],
        "threat": [],
        "timestampOverride": "event.ingested",
        "timestampOverrideFallbackDisabled": false,
        "to": "now",
        "references": [
          "https://suricata.readthedocs.io/en/suricata-6.0.9/rules/index.html"
        ],
        "version": 5,
        "exceptionsList": [
          {
            "list_id": "559aee96-104c-44bd-be68-7c4e71cd0457",
            "namespace_type": "single",
            "id": "7460dd30-8c79-11ed-94b0-654745905219",
            "type": "detection"
          }
        ],
        "type": "query",
        "language": "kuery",
        "index": [
          "auditbeat-*",
          "filebeat-*",
          "packetbeat-*",
          "winlogbeat-*",
          "logs-endpoint.events.*"
        ],
        "query": "event.kind:alert and event.module:suricata",
        "filters": []
      },
      "rule_type_id": "siem.queryRule",
      "created_by": "elastic",
      "updated_by": "elastic",
      "created_at": "2023-01-04T21:03:40.313Z",
      "updated_at": "2023-01-05T15:22:02.844Z",
      "api_key_owner": "elastic",
      "notify_when": "onActiveAlert",
      "mute_all": true,
      "muted_alert_ids": [],
      "scheduled_task_id": "43afdd40-8c73-11ed-85b3-87db925d75d3",
      "execution_status": {
        "status": "ok",
        "last_execution_date": "2023-01-05T15:22:05.901Z",
        "last_duration": 1835
      },
      "actions": []
    }
  ]
}

[brsolomon-deloitte]

Contrary to #139969 I have verified the alerts are not in cold/frozen/snapshot phase.

{
  "indices": {
    ".internal.alerts-security.alerts-default-000001": {
      "index": ".internal.alerts-security.alerts-default-000001",
      "managed": true,
      "policy": ".alerts-ilm-policy",
      "index_creation_date_millis": 1672866669277,
      "time_since_index_creation": "20.52h",
      "lifecycle_date_millis": 1672866669277,
      "age": "20.52h",
      "phase": "hot",
      "phase_time_millis": 1672866669344,
      "action": "rollover",
      "action_time_millis": 1672866669344,
      "step": "check-rollover-ready",
      "step_time_millis": 1672866669344,
      "phase_execution": {
        "policy": ".alerts-ilm-policy",
        "phase_definition": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_primary_shard_size": "50gb",
              "max_age": "30d"
            }
          }
        },
        "version": 3,
        "modified_date_in_millis": 1672843617932
      }
    }
  }
}

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[kqualters-elastic]

@brsolomon-deloitte the 3 posted log lines are likely a red herring, and the relevant log entry likely higher up in the file. Generally I've seen this when resources fail to install for some reason https://github.com/elastic/kibana/blob/main/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts#L96 perhaps try to look for the error message seen on this line

kibana/x-pack/plugins/rule_registry/server/rule_data_plugin_service/resource_installer.ts

Line 120 in b7ff354

`Error installing common resources in RuleRegistry ResourceInstaller - ${err.message}`

otherwise set the logging level to debug and restart kibana, looking for some of the log messages in that file. Error has nothing to do with the rule definition itself, and is happening before it's ever defined.

[neu5ron]

deleting preview index and restarting kibana fixed this.
will it be long term not sure, but everything finally works again.

using version 8.5.3

[neu5ron]

happens in 8.6.2 too. different cluster too.
It's all to do with the preview index & the way Kibana gets restarted either during upgrades/outages/etc or just any sort of flakey ILM issue.
Here is another related error for anyone else who ends up here.

Bulk Indexing of signals failed: RuleDataWriterInitializationError: There has been a catastrophic error trying to install index level resources for the following registration context: security. This may have been due to a non-additive change to the mappings, removal and type changes are not permitted. Full error: Error: Failure installing common resources shared between all indices. process_cluster_event_timeout_exception: [process_cluster_event_timeout_exception] Reason: failed to process cluster event (put-lifecycle-.alerts-ilm-policy) within 30s

Same fix, restart Kibana. Trying to force an ILM step did not work either.