Understanding jetstream warnings

[astelmashenko]

Hey, Community,
We are using Nats/Jetstream 2.9.19 (on alpine), three nodes like below:

 NAME                                  READY   STATUS    RESTARTS   AGE
jetstream-nats-0                      4/4     Running   0                   89m
jetstream-nats-1                      4/4     Running   0                   3d7h
jetstream-nats-2                      4/4     Running   0                  2d18h
jetstream-nats-box-6bf4bcbc4f-rssf8   2/2     Running   0          3d6h

We noticed an application failed to publish a message with timeout error and the same time a lot of messages on jetstream:

[61] 2025/02/08 15:25:27.467719 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:41.480542 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:47.005060 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:52.530745 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:25:58.054230 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:26:03.567590 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:26:40.490602 [INF] JetStream cluster new consumer leader for '$G > view_configs_change > zU6ZviET'
[61] 2025/02/08 15:27:00.083845 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:05.972497 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:11.495172 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:17.009565 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:22.535667 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:27:28.047313 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag
[61] 2025/02/08 15:28:02.647994 [WRN] JetStream stream '$G > DEFAULT_STREAM' has high message lag

We are using AWS EBS disks as storage. There were no high load or anything extraordinary.

Only restart helped to resolve that. Any thoughts on what could be the problem?
What does "has high message lag" mean?
If there is something I need to add about our system, please let me know.