Reduce the number of runtime exceptions (SecurityModuleException)

CHANGELOG.md

@@ -16,6 +16,7 @@
 - `--Xmerge-support` option remove (deprecated in 22.4.2) [#4518](https://github.com/hyperledger/besu/pull/4518)
 
 ### Additions and Improvements
+- Reduce the number of runtime exceptions (SecurityModuleException) and unnecessary executions during ECIES handshake, by trying to decrypt EIP-8 formatted messages first [#4508](https://github.com/hyperledger/besu/pull/4508).
 - Improved RLP processing of zero-length string as 0x80 [#4283](https://github.com/hyperledger/besu/pull/4283) [#4388](https://github.com/hyperledger/besu/issues/4388)
 - Increased level of detail in JSON-RPC parameter error log messages [#4510](https://github.com/hyperledger/besu/pull/4510)
 - New unstable configuration options to set the maximum time, in milliseconds, a PoS block creation jobs is allowed to run [#4519](https://github.com/hyperledger/besu/pull/4519)
@@ -24,6 +25,7 @@
 - Continuously try to build better block proposals until timeout or GetPayload is called [#4516](https://github.com/hyperledger/besu/pull/4516)
 - Avoid connecting to self when using static-nodes [#4521](https://github.com/hyperledger/besu/pull/4521)
 
+
 ### Bug Fixes
 - Corrects emission of blockadded events when rewinding during a re-org. Fix for [#4495](https://github.com/hyperledger/besu/issues/4495)
 - Always return a transaction type for pending transactions [#4364](https://github.com/hyperledger/besu/pull/4364)

ethereum/p2p/src/main/java/org/hyperledger/besu/ethereum/p2p/rlpx/handshake/ecies/ECIESHandshaker.java

@@ -189,9 +189,6 @@ public Optional<ByteBuf> handleMessage(final ByteBuf buf) throws HandshakeExcept
     try {
       // Decrypt the message with our private key.
       try {
-        bytes = EncryptedMessage.decryptMsg(bytes, nodeKey);
-        version4 = false;
-      } catch (final Exception ex) {
         // Assume new format
         final int size = bufferedBytes.readUnsignedShort();
         if (buf.writerIndex() >= size) {
@@ -203,8 +200,11 @@ public Optional<ByteBuf> handleMessage(final ByteBuf buf) throws HandshakeExcept
           bytes = EncryptedMessage.decryptMsgEIP8(encryptedMsg, nodeKey);
           version4 = true;
         } else {
-          throw new HandshakeException("Failed to decrypt handshake message", ex);
+          throw new HandshakeException("Failed to decrypt handshake message");
         }
+      } catch (final Exception ex) {
+        bytes = EncryptedMessage.decryptMsg(bytes, nodeKey);
+        version4 = false;
       }
     } catch (final InvalidCipherTextException e) {
       status.set(Handshaker.HandshakeStatus.FAILED);