[Questions] Add RabbitMQ nodes on VMs join a RabbitMQ cluster on Kubernetes

[eriktroost070]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I run RabbitMQ 4.x, the only series currently covered by community support
• I promise to provide all relevant information (versions, logs from all nodes, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

4.0.5

Erlang version used

27.2.x

Operating system (distribution) used

RHEL 8.10

How is RabbitMQ deployed?

Bitnami Helm chart

rabbitmq-diagnostics status output

Node on Kubernetes:

Status of node rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local ...
Runtime

OS PID: 198
OS: Linux
Uptime (seconds): 249884
Is under maintenance?: false
RabbitMQ version: 3.10.5
Node name: rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local
Erlang configuration: Erlang/OTP 24 [erts-12.3.2] [source] [64-bit] [smp:6:6] [ds:6:6:10] [async-threads:1] [jit]
Crypto library: OpenSSL 1.1.1n  15 Mar 2022
Erlang processes: 426 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /opt/bitnami/rabbitmq/etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_federation_management
 * rabbitmq_peer_discovery_common
 * rabbitmq_shovel_management
 * rabbitmq_shovel
 * amqp10_client
 * rabbitmq_federation
 * prometheus
 * accept
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /bitnami/rabbitmq/mnesia/rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local
Raft data directory: /bitnami/rabbitmq/mnesia/rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local/quorum/rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local

Config files

 * /opt/bitnami/rabbitmq/etc/rabbitmq/rabbitmq.conf

Log file(s)

 * /opt/bitnami/rabbitmq/var/log/rabbitmq/rabbit@te-dev-rabbitmq-migration-0.te-dev-rabbitmq-migration-headless.te-dev.svc.redacted.local_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.153 gb
Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 2.1475 gb

reserved_unallocated: 0.087 gb (56.87 %)
code: 0.0364 gb (23.83 %)
other_proc: 0.0202 gb (13.21 %)
other_system: 0.0158 gb (10.35 %)
other_ets: 0.0036 gb (2.37 %)
plugins: 0.002 gb (1.34 %)
binary: 0.0016 gb (1.05 %)
atom: 0.0016 gb (1.03 %)
mgmt_db: 3.0e-4 gb (0.21 %)
metrics: 2.0e-4 gb (0.15 %)
mnesia: 1.0e-4 gb (0.09 %)
connection_other: 0.0 gb (0.02 %)
msg_index: 0.0 gb (0.02 %)
quorum_ets: 0.0 gb (0.02 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 65439
Sockets: 0, limit: 58893

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 10.447 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 25673, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 9419, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 5673, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Node on VMs:

Status of node rabbit@rmq-d0-app0.redacted2.internal ...
Runtime

OS PID: 18
OS: Linux
Uptime (seconds): 2809
Is under maintenance?: false
RabbitMQ version: 3.10.5
Node name: rabbit@rmq-d0-app0.redacted2.internal
Erlang configuration: Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
Crypto library: OpenSSL 1.1.1q  5 Jul 2022
Erlang processes: 405 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

 * rabbitmq_prometheus
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_peer_discovery_common
 * rabbitmq_shovel_management
 * rabbitmq_shovel
 * amqp10_client
 * prometheus
 * accept
 * rabbitmq_management
 * amqp_client
 * rabbitmq_web_dispatch
 * cowboy
 * cowlib
 * rabbitmq_management_agent

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@rmq-d0-app0.redacted2.internal
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@rmq-d0-app0.redacted2.internal/quorum/rabbit@rmq-d0-app0.redacted2.internal

Config files

 * /etc/rabbitmq/rabbitmq.conf

Log file(s)

 * /var/log/rabbitmq/rabbit@rmq-d0-app0.redacted2.internal.log
 * /var/log/rabbitmq/rabbit@rmq-d0-app0.redacted2.internal_upgrade.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.1543 gb
Calculation strategy: rss
Memory high watermark setting: 0.7 of available memory, computed to: 2.8546 gb

reserved_unallocated: 0.0871 gb (56.43 %)
code: 0.0362 gb (23.46 %)
other_system: 0.0254 gb (16.46 %)
other_proc: 0.0195 gb (12.65 %)
other_ets: 0.0034 gb (2.19 %)
atom: 0.0015 gb (0.95 %)
plugins: 0.001 gb (0.64 %)
metrics: 2.0e-4 gb (0.15 %)
mgmt_db: 2.0e-4 gb (0.12 %)
binary: 2.0e-4 gb (0.11 %)
mnesia: 1.0e-4 gb (0.08 %)
msg_index: 0.0 gb (0.02 %)
quorum_ets: 0.0 gb (0.02 %)
connection_other: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
allocated_unused: 0.0 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 2, limit: 1048479
Sockets: 0, limit: 943629

Free Disk Space

Low free disk space watermark: 2.039 gb
Free disk space: 21.2763 gb

Totals

Connection count: 0
Queue count: 0
Virtual host count: 1

Listeners

Interface: [::], port: 25673, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 5673, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

Logs from node 1 (with sensitive values edited out)

N/A

Logs from node 2 (if applicable, with sensitive values edited out)

No response

Logs from node 3 (if applicable, with sensitive values edited out)

No response

rabbitmq.conf

Config on Kubernetes

## Username and password
##
default_user = user
default_pass = CHANGEME
## Clustering
##
cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
cluster_formation.k8s.host = kubernetes.default
cluster_formation.node_cleanup.interval = 10
cluster_formation.node_cleanup.only_log_warning = true
cluster_partition_handling = autoheal
load_definitions = /app/load_definition.json
# queue master locator
queue_master_locator = min-masters
# enable guest user
loopback_users.guest = false

## Prometheus metrics
##
prometheus.tcp.port = 9419
prometheus.return_per_object_metrics = true
## Memory Threshold
##
total_memory_available_override_value = 5368709120
vm_memory_high_watermark.relative = 0.4

Config on VMs:

cluster_formation.peer_discovery_backend = classic_config

cluster_formation.classic_config.nodes.0 = rabbit@rmq-d0-app0.redacted2.internal
cluster_formation.classic_config.nodes.1 = rabbit@rmq-d0-app1.redacted2.internal
cluster_formation.classic_config.nodes.2 = rabbit@rmq-d0-app2.redacted2.internal

definitions.import_backend = local_filesystem
definitions.local.path = /etc/rabbitmq/def.json

vm_memory_high_watermark.relative = 0.7
disk_free_limit.relative = 0.5

#log.file.level = debug

cluster_name = rabbit@dev0

listeners.tcp.default = 5673

Steps to deploy RabbitMQ cluster

Kubernetes RabbitMQ:
This is deployed through a Helm Chart with a Gitlab pipeline on the Kubernetes cluster.

VM RabbitMQ:
The infrastructure is deployed with Terraform and installation and configuration of RabbitMQ is done with a Ansible Playbook.

Steps to reproduce the behavior in question

Setup a RabbitMQ cluster on Kubernetes on a specific datacenter.
Setup RabbitMQ cluster on VMs on a different datacenter.
Make sure the networks of both datacenter tenants are advertised to both sides.

Now you can connect to both sides, this is needed to be able to connect the 2 RabbitMQ clusters.

And I think it is relevant that the RabbitMQ clusters/nodes on K8S do not reside on the same network as RabbitMQ on VMs. But they can connect to each other over ports 5673, 4369, 25673.

advanced.config

N/A

Application code

N/A

Kubernetes deployment file

No response

What problem are you trying to solve?

We have a RabbitMQ cluster running on Kubernetes that we need to migrate to the a cluster running on VMs on a different datacenter tenant. This migration has to be performed without downtime.

The idea was to setup RabbitMQ nodes on the VMs and have them join the RabbitMQ cluster on Kubernetes. We made sure the network connection from both sides is possible. Have all VMs node join the cluster on Kubernetes. Messages from our applications that use RabbitMQ will be send to a central Kubernetes service, which is pointing to the existing RabbitMQ cluster on Kubernetes. We setup a new service + endpoint on Kubernetes, which will point to the loadbalancer that is before the RabbitMQ cluster on VMs and thus directing the traffic to the new nodes. And then 1 by 1 remove the nodes that are running on Kubernetes when syncing of data is complete.

The problem is, I cannot find any documentation stating that this combination of RabbitMQ on Kubernetes and RabbitMQ on VMs is possible. I would assume the RabbitMQ application is not different, but the config and peer discovery are very much different.

I've tried joining 1 of the RabbitMQ node on VMs to join the RabbitMQ cluster on Kubernetes but without success:

Clustering node rabbit@rmq-d0-app0.redacted2.internal with rabbit@te-dev-rabbitmq-migration
Error: unable to perform an operation on node 'rabbit@te-dev-rabbitmq-migration.localdomain'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

 * Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
 * CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
 * Target node is not running

In addition to the diagnostics info below:

 * See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
 * Consult server logs on node rabbit@te-dev-rabbitmq-migration.localdomain
 * If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@te-dev-rabbitmq-migration.localdomain']

rabbit@te-dev-rabbitmq-migration.localdomain:
  * connected to epmd (port 4369) on te-dev-rabbitmq-migration.localdomain
  * epmd reports node 'rabbit' uses port 25673 for inter-node and CLI tool traffic
  * TCP connection succeeded but Erlang distribution failed
  * suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
  * suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
  * suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
   * suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more


Current node details:
 * node name: 'rabbitmqcli-870-rabbit@rmq-d0-app0.redacted2.internal'
 * effective user's home directory: /var/lib/rabbitmq
 * Erlang cookie hash: xxxxxxxxxxxxxx==

I've checked the Erlang cookie as that is the main culprit, but that seems to be exactly the same cookie value.

Logging does not show anything, so I'm at the moment clueless on what is going on. And if this setup is even possible.
Can anyone share if they have experience or know this could work? And are there are specific config setting that need to be checked?

[michaelklishin]

@eriktroost070 RabbitMQ nodes do not know if they are running on Kubernetes or in a VM.

The important parts are well documented in the Clustering and Cluster Formation guides:

• That node hostnames resolve for every node on every other node
• That the shared secret is identical
• That all ports are accessible

The CLI message includes several clues and several suggestions, of which you seemingly have verified just one. The node your are clustering with will log a message when the cookie is different.

Rethink the Entire Strategy

This approach to upgrades is known as "grown-then-shrink" on our team and it is explicitly recommended against unless you have a lot of experience with such migrations. It is easy to get wrong.

A Blue-Green Deployment is how clusters should be migrated from one infrastructure to the other.

Tanzu RabbitMQ users can also use the Warm Standby Replication feature for migration, even though it was not really built for that.

[eriktroost070]

Thank you for clearing several things up for me/us.

I did not interpreted the upgrade strategies as a strategy for migrations. That is a fault at my part. I read "deployment" and "upgrade" and did not really include it in my investigation, although I've read the blue-green deployment document a while back for other reasons. But the first sentence clearly should have been a clue...."Blue-green deployment is a migration technique that can also be used....".

One question about this strategy and specific federated queues.
Do interpreted this sentence correct? (it is important to us know): "The principle of federated queues is that the consumers now connected to "green" will get messages published to "blue" as long as there are no consumers in "blue" (local consumers take precedence)."

Does this mean that when there a no consumer connected to a queue on the blue cluster, message published to the queue on blue cluster will be moved to the queue on the green cluster? Even if those messages were still in the queue before the consumers switched to the green cluster (slow consumers for example)?

[michaelklishin]

If there are no consumers on queue Q1 in the blue cluster, queue federation links will move messages to the queue with the same name in the green cluster.

Queue federation links are consumers with negative priorities, so any application consumer will be prioritized over them but if none are present, the links will move messages to be consumed in the green cluster.

If there are consumers in the blue cluster, queue federation links won't have any effect since their priorities are always lower than the default, and it is extremely rare to see application developers using negative priorities, or even explicitly using consumer priorities at all.

Queue federation links do not distinguish between "old" or "new" messages, and they are not aware of other consumers.

[michaelklishin]

The best way to understand 90% of what there is to know about Blue/Green deployments is

1. Export cluster-wide definitions from the blue cluster and import them into the green one, using rabbitmqadmin v2 or rabbitmqctl, or even the management UI on both sides
2. Set up queue federation between two standalone nodes and experiment with what it does: what happens when you publish on both ends but only have consumers on one side, or both
3. Based on what you learn in the 2nd step, envision a migration strategy where you start all or some consumers in the new green cluster to let them set up the topology (publishers are often only aware of the exchanges and routing keys they use, and that's perfectly reasonable), then you would migrate the publishers, let the blue consumers drain their queues or stop them for queue federation to move the rest

That's about it. The blue cluster can be stopped when you decide to stop it. You can even go back to it if you have to for any reason.

[eriktroost070]

I really appreciate the effort you took in explaining these unknown/unclear parts for me. I'm going to setup a test environment and see if I can setup the federation and simulated a migration.

Again, thanks for all the help.