Different behavior around internode TLS on Erlang 26

[daleksic-godaddy]

Describe the bug

When Erlang 26 is used, specified config file with-ssl_dist_optfile option is not applied, therefore inter communication node on port 25672 is not configured serve TLS certificate.

Same configuration works when rabbitmq 3.12 & erlang 25 is used, but it fails in combinations rabbitmq 3.12 / erlang 26 and rabbitmq 3.13 / erlang 26.

Also, rabbitmqctl and rabbitmq-diagnostics tools fails to connect on affected versions.

Reproduction steps

PoC with steps to reproduce can be found at https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/tree/old-docker-poc?tab=readme-ov-file

Expected behavior

• inter_node_tls.config to be applied when RabbitMQ is is started with -pa $ERL_SSL_PATH -proto_dist inet_tls -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config additional flags.
• Inter communication endpoint on 25672 port should serve SSL certificate

Additional context

No response

[michaelklishin]

@daleksic-godaddy RabbitMQ does not implement TLS and is not responsible for handling of -proto_dist and friends. The runtime does/is.

Port 5672 is not supposed to be used for TLS.

The only user-facing TLS-related change that RabbitMQ had to adapt to is the new default at least one peer verification setting. RabbitMQ has retained the old default where it can control the value (this is not the case with TLS for inter-node communication), most of the changes I recall in this area have shipped in 3.13.1.

[michaelklishin]

Highly relevant erlang/otp#7497.

[lukebakken]

I've checked and, at first glance, @daleksic-godaddy's configuration file does not have that issue - https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/blob/main/docker/config/inter_node_tls.config

[lukebakken]

Hello, thanks for using RabbitMQ and for providing such a comprehensive report. I wish every RabbitMQ user were as thorough.

I have the following project to form RabbitMQ clusters, and this branch will set one up using TLS -

https://github.com/lukebakken/docker-rabbitmq-cluster/tree/tls

I'm making sure it still works as intended, but you should be able to compare what I do with what you do to figure out the difference.

[lukebakken]

Hmm, something isn't quite right with your project, because running rabbitmqctl status using the rabbitmq312-erlang25 service isn't working, either. I'm investigating.

[lukebakken]

@daleksic-godaddy I have spent probably more time today than I should trying to figure out why your project doesn't work out-of-the-box in my docker environment. I had to make the following changes so that the make test_erlang25_rmq312 target works once the rabbitmq312-erlang25 service is running:

daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc#1

If I docker compose exec into the rabbitmq312-erlang25 container, and run rabbitmqctl status, it ONLY works if I disable peer verification on the inter-node TLS client:

https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/pull/1/files#diff-c2d90e5845596a73ed9aff5c445e389d3233d90e887474a8dcbaf4ece784825eR18

I can't explain this behavior.

One major difference between my working project and yours is that I use the official RabbitMQ docker image.

I don't have the time to continue investigating this issue as I have to focus support efforts on RabbitMQ users with support contracts. I will return to this as my time and interest allow. Thanks again for the comprehensive report.

For what it's worth, this is what I see when I clone your project as-is and run make:
rabbitmq-server-11074-make.log

2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> BOOT FAILED
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> ===========
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0> Error during startup: {error,failed_to_prepare_configuration}
2024-04-24 17:25:20.391515+00:00 [error] <0.135.0>
Error during startup: {error,failed_to_prepare_configuration}

2024-04-24 17:25:20.491289+00:00 [error] <0.131.0> ssl_options.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.512177+00:00 [error] <0.131.0> ssl_options.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514094+00:00 [error] <0.131.0> management.ssl.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514195+00:00 [error] <0.131.0> management.ssl.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514466+00:00 [error] <0.131.0> Error preparing configuration in phase validation:
2024-04-24 17:25:20.514500+00:00 [error] <0.131.0>   - ssl_options.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514561+00:00 [error] <0.131.0>   - ssl_options.keyfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514620+00:00 [error] <0.131.0>   - management.ssl.certfile invalid, file does not exist or cannot be read by the node
2024-04-24 17:25:20.514673+00:00 [error] <0.131.0>   - management.ssl.keyfile invalid, file does not exist or cannot be read by the node

I don't see how it could work in your environment. Please review my PR.

[daleksic-godaddy]

@lukebakken Thanks for in-depth analysis.

Ok, this is related to how docker mounts files inside container (owned by root) - my mistake 😅. I overlooked it, as I was able to replicate non-serving cert issue when checking with openssl command. Our actual setup is not using containers, we are on Ubuntu 22.04 VMs. This was my quick attempt to replicate issue using lightweight PoC environment instead full blown VMs. That's why I didn't used official rabbimtq docker images, instead I've used custom one which mimics our ansible provisioning.

I will check once more configuration and examples that you provided, and properly replicate issue if still persists.

[daleksic-godaddy]

@lukebakken I was able to fully replicate issue using Vagrant local environment, and also find the cause 🎉 .

Here is the new PoC enviroment https://github.com/daleksic-godaddy/rabbitmq-erlang-26-ssl-dist-optfile-issue-poc/tree/60f91070f6d8bc6b0341e632b73361cf16dc8648

In a nutshell, difference is due to how ssl options are handled in Erlang 26. It seems that excess options are not ignored anymore in erlang26, therefore:

[
  {server, [
    {cacertfile, "/etc/rabbitmq/tls/rabbitmq_server_wildcard.ca"},
    {certfile,   "/etc/rabbitmq/tls/rabbitmq_server_wildcard.crt"},
    {keyfile,    "/etc/rabbitmq/tls/rabbitmq_server_wildcard.key"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true},
    {customize_hostname_check, [
      {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]},
  {client, [
    {cacertfile, "/etc/rabbitmq/tls/rabbitmq_client_wildcard.ca"},
    {certfile,   "/etc/rabbitmq/tls/rabbitmq_client_wildcard.crt"},
    {keyfile,    "/etc/rabbitmq/tls/rabbitmq_client_wildcard.key"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {customize_hostname_check, [
      {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]}
].

This config will work fine with Erlang 25, but it would fail on Erlang 26 as customize_hostname_check is only valid for client section (clue 01 & clue 02).

This then leads that server section is not loaded at all, therefore ssl certificate is not served on 25672 (internode communication) port (as seen in the README with client_variant: "wildcard_cert_both_customize" set in the Vagrant PoC. Didn't have time to deeply analyze Erlang code, this is based on my observation and testing.

Concerning part is that this is breaking change for folks that configured theirs setup with customize_hostname_check in both section, and want to upgrade to Erlang 26 & newer RabbitMQ versions. Also, a found few examples like this one #10398 (comment) that contains example where customize_hostname_check is specified in both places. I've also found similar one where {fail_if_no_peer_cert, true} was also included in both client and server section and it was failing to to same reason as here, but can't find link at the moment.

I would strongly suggest to add/update documentation to include:

1. note about new behavior in Erlang 26 that may lead to breaking setups
2. note about ensuring that all configuration options are only put in appropriate sections / remove excess

Also, rabbitmq/rabbitmq-website#1791 to be resolved, would be highly appreciated, as I was also affected same issue where customize_hostname_check is not metioned as required for wildcard certs in the docs.

[lukebakken]

Thanks for following up.

Yep, this caught me as well last year when I started using Erlang 26.

Please see the following:

• Distributed Erlang over TLS does not work correctly using OTP 26 erlang/otp#7497
• Distributed Erlang over TLS does not work correctly using OTP 26 erlang/otp#7497 (comment)
• Remove invalid TLS settings rabbitmq-website#1687

Our docs have been updated for a while to reflect fail_if_no_peer_cert, but not customize_hostname_check, because it is very rare for people to use wildcard certs for clusters.

If someone from the community would like to update the docs, a pull request would be greatly appreciated.

[michaelklishin]

And here's where the website source code lives. Note that in this case contributors should make sure to update both main and the versioned_docs 3.13.x versions.

[manirajp]

Could someone please help? I am using RabbitMQ version 3.13.3 and Erlang version 26.2.5.1.
I am facing an issue where the rabbitmqctl status command is not working. I am getting the following error with the current rabbitmq-env.conf configuration

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -proto_dist inet_tls -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"

RABBITMQ_CTL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"

RABBITMQ_PLUGINS_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tls -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"_

---

[rabbit@node]$ /usr/lib/rabbitmq/bin/rabbitmqctl status
Error: unable to perform an operation on node 'rabbit@node'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

• Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
• CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
• Target node is not running

In addition to the diagnostics info below:

• See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
• Consult server logs on node rabbit@node
• If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS

attempted to contact: ['rabbit@node']

rabbit@node:

• connected to epmd (port 4369) on node
• epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
• TCP connection succeeded but Erlang distribution failed
• suggestion: check if the Erlang cookie is identical for all server nodes and CLI tools
• suggestion: check if all server nodes and CLI tools use consistent hostnames when addressing each other
• suggestion: check if inter-node connections may be configured to use TLS. If so, all nodes and CLI tools must do that
• suggestion: see the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more

Current node details:

• node name: 'rabbitmqcli-846-rabbit@node'
• effective user's home directory: /data/rabbitmq
• Erlang cookie hash: S2emRi+iF1PhWmw3i4myjg==

---

But when I changed the -proto_dist inet_tcp, it started working.
However, after going through the documentation, I found that the inet_tcp option is not very secure, so I still need to enable the inet_tls option

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tcp -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"

RABBITMQ_CTL_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tcp -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"

RABBITMQ_PLUGINS_ERL_ARGS="-pa ${ERL_SSL_PATH} -proto_dist inet_tcp -ssl_dist_opt server_certfile /data/rabbitmq/certs/server/rabbit.pem -ssl_dist_opt server_secure_renegotiate true -ssl_dist_opt client_secure_renegotiate true"

Could you guys please help how can resolve this issue to use the TLS [tried all the cipher addition in rabbitmq-env.conf but still 'rabbitmqctl status' cli failing]

[michaelklishin]

@manirajp we do not appreciate existing discussions being hijacked with new questions.

RabbitMQ 3.13 is out of community support. Our Community support policy clearly states that we will not debug TLS for you unless you are a paying customer or a regular contributor.

Inter-node Clustering and CLI guides both mention that all nodes and CLI tools must be configured to use TLS, or they will fail to connect to each other. A TLS-enabled client cannot connect to a non-TLS port and vice versa.

[michaelklishin]

#11074 (reply in thread) mentioned earlier above seems to be a fairly universal answer to what this discussion was about: the TLS client default changes in Erlang 26+ (where peer certificate chain verification is now enabled by default).

It can still be disabled, see Erlang's inter-node distribution connections with TLS and search for "verify".

Team RabbitMQ's general guidelines on troubleshooting TLS can also be used with inter-node connections.

[michaelklishin]

@manirajp we do not appreciate existing discussions being hijacked with new questions.

RabbitMQ 3.13 is out of community support. Our Community support policy clearly states that we will not debug TLS for you unless you are a paying customer or a regular contributor.