system_memory_high_watermark after stopping and starting a k8s cluster - 4.0.3

[ewencodes]

Describe the bug

Hi,
I encounter an issue on my RabbitMQ cluster deployed on an kubernetes cluster.

After migrating from 3.13 to 4.0 and after restarting the kubernetes cluster, this issue appear and make my cluster in stuck mode.

I try increasing the RAM and the CPU allocated to the RabbitMQ cluster to allow it to start but without any success..

Logs :

rabbitmq 08:49:04.63 INFO  ==>
rabbitmq 08:49:04.64 INFO  ==> Welcome to the Bitnami rabbitmq container
rabbitmq 08:49:04.64 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
rabbitmq 08:49:04.64 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
rabbitmq 08:49:04.64 INFO  ==> Upgrade to Tanzu Application Catalog for production environments to access custom-configured and pre-packaged software components. Gain enhanced features, including Software Bill of Materials (SBOM), CVE scan result reports, and VEX documents. To learn more, visit https://bitnami.com/enterprise
rabbitmq 08:49:04.65 INFO  ==>
rabbitmq 08:49:04.65 INFO  ==> ** Starting RabbitMQ setup **
rabbitmq 08:49:04.69 INFO  ==> Validating settings in RABBITMQ_* env vars..
rabbitmq 08:49:04.70 WARN  ==> A definition file with "users" was found. The RABBITMQ_SECURE_PASSWORD environment variables will be ignored.
rabbitmq 08:49:04.72 INFO  ==> Initializing RabbitMQ...
rabbitmq 08:49:04.73 DEBUG ==> Creating environment file...
rabbitmq 08:49:04.73 DEBUG ==> Creating enabled_plugins file...
rabbitmq 08:49:04.74 DEBUG ==> Creating Erlang cookie...
rabbitmq 08:49:04.75 DEBUG ==> Ensuring expected directories/files exist...
rabbitmq 08:49:04.79 INFO  ==> Persisted data detected. Restoring...
rabbitmq 08:49:04.80 WARN  ==> Forcing node to start...
rabbitmq 08:49:07.12 INFO  ==> No custom scripts in /docker-entrypoint-initdb.d

rabbitmq 08:49:07.12 INFO  ==> ** RabbitMQ setup finished! **
rabbitmq 08:49:07.16 INFO  ==> ** Starting RabbitMQ **
=INFO REPORT==== 11-Dec-2024::08:49:07.854738 ===
    alarm_handler: {set,{system_memory_high_watermark,[]}}
2024-12-11 08:49:10.600470+00:00 [notice] <0.44.0> Application syslog exited with reason: stopped
2024-12-11 08:49:10.606524+00:00 [notice] <0.254.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
2024-12-11 08:49:10.607158+00:00 [notice] <0.254.0> Logging: configured log handlers are now ACTIVE
2024-12-11 08:49:10.615374+00:00 [info] <0.254.0> ra: starting system quorum_queues
2024-12-11 08:49:10.615443+00:00 [info] <0.254.0> starting Ra system: quorum_queues in directory: /bitnami/rabbitmq/mnesia/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local/quorum/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local
2024-12-11 08:49:10.683999+00:00 [info] <0.267.0> ra system 'quorum_queues' running pre init for 25 registered servers
2024-12-11 08:49:10.835161+00:00 [info] <0.268.0> ra: meta data store initialised for system quorum_queues. 25 record(s) recovered
2024-12-11 08:49:10.869397+00:00 [notice] <0.273.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
2024-12-11 08:49:10.894242+00:00 [info] <0.275.0> ra_system_recover: ra system 'quorum_queues' server recovery strategy rabbit_quorum_queue:system_recover
2024-12-11 08:49:10.894369+00:00 [info] <0.275.0> [rabbit_quorum_queue:system_recover/1] rabbit not booted, skipping queue recovery
2024-12-11 08:49:10.894576+00:00 [info] <0.254.0> ra: starting system coordination
2024-12-11 08:49:10.894620+00:00 [info] <0.254.0> starting Ra system: coordination in directory: /bitnami/rabbitmq/mnesia/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local/coordination/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local
2024-12-11 08:49:10.896100+00:00 [info] <0.281.0> ra system 'coordination' running pre init for 1 registered servers
2024-12-11 08:49:10.901823+00:00 [info] <0.282.0> ra: meta data store initialised for system coordination. 1 record(s) recovered
2024-12-11 08:49:10.902039+00:00 [notice] <0.287.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
2024-12-11 08:49:10.913196+00:00 [info] <0.254.0> ra: starting system coordination
2024-12-11 08:49:10.913301+00:00 [info] <0.254.0> starting Ra system: coordination in directory: /bitnami/rabbitmq/mnesia/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local/coordination/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local
2024-12-11 08:49:11.210968+00:00 [notice] <0.292.0> RabbitMQ metadata store: candidate -> leader in term: 114 machine version: 1
Mnesia('rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local'): previous_log repaired, lost 3509 bad bytes
Mnesia('rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local'): previous_log repaired, lost 7 bad bytes
Mnesia('rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local'): previous_log repaired, lost 3509 bad bytes
2024-12-11 08:49:11.488914+00:00 [notice] <0.317.0> disk_log: repairing "/bitnami/rabbitmq/mnesia/rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local/rabbit_durable_queue.DCL" ...
2024-12-11 08:49:11.488914+00:00 [notice] <0.317.0>

Reproduction steps

No reproduction.

Expected behavior

My RabbitMQ cluster to restart.

Additional context

Did you know anything that can unblock me ?

Thanks a lot.

[lukebakken]

Hello, thanks for using RabbitMQ.

What do you expect us to do with the very limited information you have provided? Nobody on Team RabbitMQ is telepathic.

[ewencodes]

Hi @lukebakken, sorry for the lack of information provided on the issue but its all i have. I don't know what to provide for debugging since i'm using a simple setup of RabbitMQ.

[michaelklishin]

@ewencodes we cannot suggest anything beyond this doc guide that explains how to use rabbitmq-diagnostics or other available tools to understand what consumes memory on your node. Prometheus and Grafana dashboards can help you collect and chart effectively the same kind of information over time.

A few lines from early during node boot tell us nothing about your workload and we do not guess on this team.

[ewencodes]

thanks for all informations provided in your response @michaelklishin.

i try using rabbitmq-diagnostics report to generate a report but without any success because my cluster is stuck at booting state :

I have no name!@sw-common-rabbitmq-0:/$ rabbitmq-diagnostics report
Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'.
Arguments given:
        report

Usage

rabbitmq-diagnostics [--node <node>] [--longnames] [--quiet] report

but, when i use rabbitmq-diagnostics is_booting i have an output :

I have no name!@sw-common-rabbitmq-0:/$ rabbitmq-diagnostics is_booting
Asking node rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local for its boot status ...
RabbitMQ on node rabbit@sw-common-rabbitmq-0.sw-common-rabbitmq-headless.core.svc.cluster.local is booting

[michaelklishin]

@ewencodes "booting" here almost certainly means waiting for peers to come online, this is a classic example on K8S that is documented in detail in multiple guides [1][2][3]

1. https://www.rabbitmq.com/docs/cluster-formation#peer-discovery-k8s
2. https://www.rabbitmq.com/docs/clustering#restarting-schema-sync
3. https://www.rabbitmq.com/docs/clustering#restarting-readiness-probes

[michaelklishin]

Management UI provides the same breakdown (visually, not as a percentage table IIRC) on the node page. Click on the node name in the list of nodes, then scroll down.

[michaelklishin]

#12885 could potentially be related if there are very fast publishers that being hammering a freshly booted node. It has all the context, preview release artifacts to try, and so on. And if that indeed is the behavior (which is anyone's guess), 4.0.5 will ship next week.

[ewencodes]

thanks for the support @michaelklishin, i will try to upgrade to 4.0.5-alpha.ebfeab82.