Trying to implement TLS inter-node communication, "bad_cert,hostname_check_failed"

[papan2012]

Greetings all

I've been trying to get the rabbitmq cluster running using the TLS inter-node communication for three-node-cluster

I'm using tls_gen to generate the SSL certificates on the domain "*.rabbitmq", and all the tests using openssl are working fine:
openssl s_server -accept 8443 -cert server_certificate.pem -key server_key.pem -CAfile ca_cert.pem
openssl s_client -connect localhost:8443 -cert 'client_certificate.pem' -key 'client_key.pem' -CAfile ca_cert.pem -verify 8 -verify_hostname *.rabbitmq

RESULT:

    Start Time: 1706018367
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0

Executing s_client on the server, testing the connection to ports 5671 and 25672 results in the successful verification
openssl s_client -connect localhost:25672 -cert 'client_certificate.pem' -key 'client_key.pem' -CAfile ca_cert.pem -verify 8 -verify_hostname *.rabbitmq

---
SSL handshake has read 3862 bytes and written 3852 bytes
Verification: OK
Verified peername: *.rabbitmq
---

But rabbitmq cluster is not starting properly.
Only one node is up, with the following errors in the log

2024-01-23 14:58:26.759488+01:00 [notice] <0.412.0> TLS client: In state certify at ssl_handshake.erl:2113 generated CLIENT ALERT: Fatal - 
Handshake Failure
2024-01-23 14:58:26.759488+01:00 [notice] <0.412.0>  - {bad_cert,hostname_check_failed}

I see the same messages on other nodes, who fail to verify and fail to start because of those TLS verification failures.

Any help in explaining to me what I'm doing wrong while configuring this would be much appreciated.

I'm pasting rabbitmq configuration files bellow:

rabbitmq.conf

cluster_name=rabbittest
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_classic_config

auth_backends.1   = rabbit_auth_backend_internal
auth_mechanisms.1 = PLAIN
auth_mechanisms.2 = EXTERNAL

cluster_formation.classic_config.nodes.1 = rabbitmq@rbn01.rabbitmq
cluster_formation.classic_config.nodes.2 = rabbitmq@rbn02.rabbitmq
cluster_formation.classic_config.nodes.3 = rabbitmq@rbn03.rabbitmq

listeners.ssl.default = 5671

ssl_options.cacertfile = /etc/rabbitmq/ssl/ca_certificate.pem
ssl_options.certfile   = /etc/rabbitmq/ssl/server_certificate.pem
ssl_options.keyfile    = /etc/rabbitmq/ssl/server_key.pem
ssl_options.verify     = verify_peer
ssl_options.fail_if_no_peer_cert = true

rabbitmq-env.conf

RABBITMQ_USE_LONGNAME=true
NODENAME=rabbitmq

NODE_IP_ADDRESS=192.168.1.5

NODE_PORT=5672

ERL_SSL_PATH=/usr/lib/erlang/lib/ssl-10.8.7/ebin

SERVER_ADDITIONAL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"

RABBITMQ_CTL_ERL_ARGS="-pa $ERL_SSL_PATH
  -proto_dist inet_tls
  -ssl_dist_optfile /etc/rabbitmq/inter_node_tls.config"

inter_node_tls.config

[
  {server, [
    {cacertfile, "/etc/rabbitmq/ssl/ca_cert.pem"},
    {certfile,   "/etc/rabbitmq/ssl/server_certificate.pem"},
    {keyfile,    "/etc/rabbitmq/ssl/server_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true}
  ]},
  {client, [
    {cacertfile, "/etc/rabbitmq/ssl/ca_cert.pem"},
    {certfile,   "/etc/rabbitmq/ssl/client_certificate.pem"},
    {keyfile,    "/etc/rabbitmq/ssl/client_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer}
  ]}
].

[lukebakken]

Thank you for the mostly complete problem report. It is always good to include software versions when reporting issues. In this case, knowing the Erlang and RabbitMQ version would be nice.

You are using wildcard certificates, which require "special handling" when used with Erlang. Most RabbitMQ users do not use wildcard certs.

If you do this search -

https://www.google.com/search?q=rabbitmq+erlang+wildcard+certificate

...one of the hits is this discussion:

https://groups.google.com/g/rabbitmq-users/c/ylwgLhdTGEw

...which leads to this issue:

vernemq/vernemq#1485

So, your inter_node_tls.config file should be this:

[
  {server, [
    {cacertfile, "/etc/rabbitmq/ssl/ca_cert.pem"},
    {certfile,   "/etc/rabbitmq/ssl/server_certificate.pem"},
    {keyfile,    "/etc/rabbitmq/ssl/server_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {fail_if_no_peer_cert, true},
    {customize_hostname_check, [
        {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]},
  {client, [
    {cacertfile, "/etc/rabbitmq/ssl/ca_cert.pem"},
    {certfile,   "/etc/rabbitmq/ssl/client_certificate.pem"},
    {keyfile,    "/etc/rabbitmq/ssl/client_key.pem"},
    {secure_renegotiate, true},
    {verify, verify_peer},
    {customize_hostname_check, [
        {match_fun, public_key:pkix_verify_hostname_match_fun(https)}
    ]}
  ]}
].

I will make a note to document this better on our website: rabbitmq/rabbitmq-website#1791

Please confirm that the above configuration addresses your issue. Thanks again!

[daleksic-godaddy]

For folks that stumble on this issue, customize_hostname_check in server section breaks configuration with Erlang 26: #11074 (comment)

[papan2012]

Thank you very much, lukebakken, this configuration you provided solved the issue and now nodes are forming the cluster.

[lukebakken]

I appreciate the follow-up!