From 85eff0c67eff2ebf9da4a9bb3d489bdfd8826df8 Mon Sep 17 00:00:00 2001 From: Jacek Sieka Date: Thu, 12 Dec 2024 14:58:02 +0100 Subject: [PATCH 01/14] Clarify gossip limits In the gossip specification, the `GOSSIP_MAX_SIZE` constant is specified for the uncompressed payload size in the gossipsub message. This PR clarifies how this limit applies to the various fields of the gossipsub message and provides additional limits derived from it that allow clients to more aggressively discard messages. In particular, clients are allowed to impose more strict limits on topics such as attestation and aggregates - an `Attestation` for example takes no more than `~228` bytes (to be verified!), far below the 10mb limit, though implicitly clients should already see these limits imposed as rejections by their SSZ decoder - this clarification mainly highlights the possibilty to perform this check earlier in the process. --- specs/phase0/p2p-interface.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 396e4671b8..5fd2771829 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -268,12 +268,22 @@ This defines both the type of data being sent on the topic and how the data fiel - `Encoding` - the encoding strategy describes a specific representation of bytes that will be transmitted over the wire. See the [Encodings](#Encodings) section for further details. +Clients MUST reject messages with unknown topic. + *Note*: `ForkDigestValue` is composed of values that are not known until the genesis block/state are available. Due to this, clients SHOULD NOT subscribe to gossipsub topics until these genesis values are known. -Each gossipsub [message](https://github.com/libp2p/go-libp2p-pubsub/blob/master/pb/rpc.proto#L17-L24) has a maximum size of `GOSSIP_MAX_SIZE`. -Clients MUST reject (fail validation) messages that are over this size limit. -Likewise, clients MUST NOT emit or propagate messages larger than this limit. +The uncompressed payload in the [`data`](https://github.com/libp2p/go-libp2p-pubsub/blob/c06df2f9a38e9382e644b241adf0e96e5ca00955/pb/rpc.proto#L19) +must have has a size no greater than `GOSSIP_MAX_SIZE`. + +After compression, the payload in the `data` field must have a size no greater than +`32 + GOSSIP_MAX_SIZE + GOSSIP_MAX_SIZE / 6` (rounded down), as given by the +[snappy maximum compressed size function](https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47). + +Clients MUST reject (fail validation) messages with payloads that are over these size limits. +Likewise, clients MUST NOT emit or propagate messages larger than these limits. + +Clients MAY use [size bounds derived from the payload SSZ type](#what-are-ssz-type-size-bounds) to determine the payload size limit, when this size is lower than `GOSSIP_MAX_SIZE`. The optional `from` (1), `seqno` (3), `signature` (5) and `key` (6) protobuf fields are omitted from the message, since messages are identified by content, anonymous, and signed where necessary in the application layer. @@ -288,6 +298,10 @@ The `message-id` of a gossipsub message MUST be the following 20 byte value comp the concatenation of `MESSAGE_DOMAIN_INVALID_SNAPPY` with the raw message data, i.e. `SHA256(MESSAGE_DOMAIN_INVALID_SNAPPY + message.data)[:20]`. +Where relevant, clients MUST reject messages with `message-id` sizes other than 20 bytes. + +Clients MAY reject messages whose protobuf-encoded size exceeds the maximum possible size based on the limits above. + *Note*: The above logic handles two exceptional cases: (1) multiple snappy `data` can decompress to the same value, and (2) some message `data` can fail to snappy decompress altogether. From 022bb22c777dd9b050c8eb6a1686101bfe2f5013 Mon Sep 17 00:00:00 2001 From: Jacek Sieka Date: Tue, 17 Dec 2024 11:31:35 +0100 Subject: [PATCH 02/14] Use single constant for gossip/req/resp, clarify encoded sizes --- specs/phase0/p2p-interface.md | 70 +++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 23 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 5fd2771829..29624e4fa8 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -193,11 +193,10 @@ This section outlines configurations that are used in this spec. | Name | Value | Description | |---|---|---| -| `GOSSIP_MAX_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed gossip messages. | +| `MAX_PAYLOAD_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed payload in gossipsub messages / RPC chunks. | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | | `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on a subnet subscription (~27 hours) | | `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | -| `MAX_CHUNK_SIZE` | `10 * 2**20` (=10485760, 10 MiB) | The maximum allowed size of uncompressed req/resp chunked responses. | | `ATTESTATION_PROPAGATION_SLOT_RANGE` | `32` | The maximum number of slots during which an attestation can be propagated. | | `MAXIMUM_GOSSIP_CLOCK_DISPARITY` | `500` | The maximum **milliseconds** of clock disparity assumed between honest nodes. | | `MESSAGE_DOMAIN_INVALID_SNAPPY` | `DomainType('0x00000000')` | 4-byte domain for gossip message-id isolation of *invalid* snappy messages | @@ -229,6 +228,21 @@ Where is entirely independent of the ENR sequence number, and will in most cases be out of sync with the ENR sequence number. +### Maximum message sizes + +Maximum message sizes are derived from the maximum payload size that the network can carry according to the following functions: + +```python +def max_compressed_len(n): + # Worst-case compressed length for a given payload of size n when using snappy + # https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47 + return int(32 + n + n / 6) + +def max_message_size(): + # Allow 1024 bytes for framing and encoding overhead but at least 1MB in case MAX_PAYLOAD_SIZE is small. + return max(max_compressed_len(MAX_PAYLOAD_SIZE) + 1024, 1024*1024) +``` + ### The gossip domain: gossipsub Clients MUST support the [gossipsub v1](https://github.com/libp2p/specs/blob/master/pubsub/gossipsub/gossipsub-v1.0.md) libp2p Protocol @@ -273,18 +287,6 @@ Clients MUST reject messages with unknown topic. *Note*: `ForkDigestValue` is composed of values that are not known until the genesis block/state are available. Due to this, clients SHOULD NOT subscribe to gossipsub topics until these genesis values are known. -The uncompressed payload in the [`data`](https://github.com/libp2p/go-libp2p-pubsub/blob/c06df2f9a38e9382e644b241adf0e96e5ca00955/pb/rpc.proto#L19) -must have has a size no greater than `GOSSIP_MAX_SIZE`. - -After compression, the payload in the `data` field must have a size no greater than -`32 + GOSSIP_MAX_SIZE + GOSSIP_MAX_SIZE / 6` (rounded down), as given by the -[snappy maximum compressed size function](https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47). - -Clients MUST reject (fail validation) messages with payloads that are over these size limits. -Likewise, clients MUST NOT emit or propagate messages larger than these limits. - -Clients MAY use [size bounds derived from the payload SSZ type](#what-are-ssz-type-size-bounds) to determine the payload size limit, when this size is lower than `GOSSIP_MAX_SIZE`. - The optional `from` (1), `seqno` (3), `signature` (5) and `key` (6) protobuf fields are omitted from the message, since messages are identified by content, anonymous, and signed where necessary in the application layer. Starting from Gossipsub v1.1, clients MUST enforce this by applying the `StrictNoSign` @@ -300,8 +302,6 @@ The `message-id` of a gossipsub message MUST be the following 20 byte value comp Where relevant, clients MUST reject messages with `message-id` sizes other than 20 bytes. -Clients MAY reject messages whose protobuf-encoded size exceeds the maximum possible size based on the limits above. - *Note*: The above logic handles two exceptional cases: (1) multiple snappy `data` can decompress to the same value, and (2) some message `data` can fail to snappy decompress altogether. @@ -516,6 +516,16 @@ so [basic snappy block compression](https://github.com/google/snappy/blob/master Implementations MUST use a single encoding for gossip. Changing an encoding will require coordination between participating implementations. +#### Gossipsub size limits + +Size limits are placed both on the [`RPCMsg`](https://github.com/libp2p/specs/blob/b5f7fce29b32d4c7d0efe37b019936a11e5db872/pubsub/README.md#the-rpc) frame as well as the encoded payload in each [`Message`](https://github.com/libp2p/specs/blob/b5f7fce29b32d4c7d0efe37b019936a11e5db872/pubsub/README.md#the-message). + +Clients MUST reject and MUST NOT emit or propagate messages whose size exceed the following limits: + +* the size of the encoded `RPCMsg`, including control messages and framing, must not exceed `max_message_size()` +* the size of the compressed payload in the `Message.data` field must not exceed `max_compressed_len(MAX_PAYLOAD_SIZE)`. +* the size of the uncompressed payload must not exceed `MAX_PAYLOAD_SIZE` or the [type-specific SSZ bound](#what-are-ssz-type-size-bounds), whichever is lower. + ### The Req/Resp domain #### Protocol identification @@ -565,7 +575,7 @@ All other response types (non-Lists) send a single `response_chunk`. For both `request`s and `response`s, the `encoding-dependent-header` MUST be valid, and the `encoded-payload` must be valid within the constraints of the `encoding-dependent-header`. This includes type-specific bounds on payload size for some encoding strategies. -Regardless of these type specific bounds, a global maximum uncompressed byte size of `MAX_CHUNK_SIZE` MUST be applied to all method response chunks. +Regardless of these type specific bounds, a global maximum uncompressed byte size of `MAX_PAYLOAD_SIZE` MUST be applied to all method response chunks. Clients MUST ensure that lengths are within these bounds; if not, they SHOULD reset the stream immediately. Clients tracking peer reputation MAY decrement the score of the misbehaving peer under this circumstance. @@ -679,15 +689,13 @@ When snappy is applied, it can be passed through a buffered Snappy reader to dec Before reading the payload, the header MUST be validated: - The unsigned protobuf varint used for the length-prefix MUST not be longer than 10 bytes, which is sufficient for any `uint64`. -- The length-prefix is within the expected [size bounds derived from the payload SSZ type](#what-are-ssz-type-size-bounds). +- The length-prefix is within the expected [size bounds derived from the payload SSZ type](#what-are-ssz-type-size-bounds) or `MAX_PAYLOAD_SIZE`, whichever is smaller. After reading a valid header, the payload MAY be read, while maintaining the size constraints from the header. -A reader SHOULD NOT read more than `max_encoded_len(n)` bytes after reading the SSZ length-prefix `n` from the header. -- For `ssz_snappy` this is: `32 + n + n // 6`. - This is considered the [worst-case compression result](https://github.com/google/snappy/blob/537f4ad6240e586970fe554614542e9717df7902/snappy.cc#L98) by Snappy. +A reader MUST NOT read more than `max_compressed_len(n)` bytes after reading the SSZ length-prefix `n` from the header. -A reader SHOULD consider the following cases as invalid input: +A reader MUST consider the following cases as invalid input: - Any remaining bytes, after having read the `n` SSZ bytes. An EOF is expected if more bytes are read than required. - An early EOF, before fully reading the declared length-prefix worth of SSZ bytes. @@ -1444,7 +1452,7 @@ Nevertheless, in the case of `ssz_snappy`, messages are still length-prefixed wi * Alignment with protocols like gRPC over HTTP/2 that prefix with length * Sanity checking of message length, and enabling much stricter message length limiting based on SSZ type information, to provide even more DOS protection than the global message length already does. - E.g. a small `Status` message does not nearly require `MAX_CHUNK_SIZE` bytes. + E.g. a small `Status` message does not nearly require `MAX_PAYLOAD_SIZE` bytes. [Protobuf varint](https://developers.google.com/protocol-buffers/docs/encoding#varints) is an efficient technique to encode variable-length (unsigned here) ints. Instead of reserving a fixed-size field of as many bytes as necessary to convey the maximum possible value, this field is elastic in exchange for 1-bit overhead per byte. @@ -1693,6 +1701,22 @@ Other types are static, they have a fixed size: no dynamic-length content is inv For reference, the type bounds can be computed ahead of time, [as per this example](https://gist.github.com/protolambda/db75c7faa1e94f2464787a480e5d613e). It is advisable to derive these lengths from the SSZ type definitions in use, to ensure that version changes do not cause out-of-sync type bounds. +#### Why is the message size defined in terms of application payload? + +When transmitting messages over gossipsub and / or req/resp, we want to ensure that the same payload sizes are supported no matter the underlying transport, decoupling the consensus layer from libp2p-induced overhead and the particular transmission strategy. + +To derive "encoded size limits" from desired application sizes we take into account snappy compression and framing overhead. + +In the case of gossipsub, the protocol supports sending multiple application payloads as well as mixing application data with control messages in each gossipsub frame - the limit is set such that at least one max-sized application-level message together with a small amount (1kb) of gossipsub overhead is allowed - implementations are free to pack multiple smaller application messages into a single gossipsub frame, and / or combine it with control messages as they see fit. + +The limit is set on the uncompressed payload size in particular to protect against decompression bombs - although + +#### Why is there a limit on message sizes at all? + +The message size limit protects against several forms of DoS and network-based amplification attacks and provide upper bounds for resource (network, memory) usage in the client based on protocol requirements to decode, buffer, cache, store and re-transmit messages which in turn translate into performance and protection tradeoffs, ensuring capacity to handle worst cases during recovery from network instability. + +In particular, blocks which at the time of writing is the only message type without a practical SSZ-derived upper bound on size cannot be fully verified synchronously as part of gossipsub validity checks meaning that there exist cases where invalid messages signed by a validator may be amplified by the network. + ## libp2p implementations matrix This section will soon contain a matrix showing the maturity/state of the libp2p features required From 44ab11d1551dc5e3d017d9a98ef37d339a437d18 Mon Sep 17 00:00:00 2001 From: Jacek Sieka Date: Tue, 17 Dec 2024 11:44:26 +0100 Subject: [PATCH 03/14] doctoc --- specs/phase0/p2p-interface.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 29624e4fa8..8bc05844c1 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -16,6 +16,7 @@ - [Constants](#constants) - [Configuration](#configuration) - [MetaData](#metadata) + - [Maximum message sizes](#maximum-message-sizes) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) @@ -28,6 +29,7 @@ - [`beacon_attestation_{subnet_id}`](#beacon_attestation_subnet_id) - [Attestations and Aggregation](#attestations-and-aggregation) - [Encodings](#encodings) + - [Gossipsub size limits](#gossipsub-size-limits) - [The Req/Resp domain](#the-reqresp-domain) - [Protocol identification](#protocol-identification) - [Req/Resp interaction](#reqresp-interaction) @@ -102,6 +104,8 @@ - [Why are we using Snappy for compression?](#why-are-we-using-snappy-for-compression) - [Can I get access to unencrypted bytes on the wire for debugging purposes?](#can-i-get-access-to-unencrypted-bytes-on-the-wire-for-debugging-purposes) - [What are SSZ type size bounds?](#what-are-ssz-type-size-bounds) + - [Why is the message size defined in terms of application payload?](#why-is-the-message-size-defined-in-terms-of-application-payload) + - [Why is there a limit on message sizes at all?](#why-is-there-a-limit-on-message-sizes-at-all) - [libp2p implementations matrix](#libp2p-implementations-matrix) @@ -522,7 +526,7 @@ Size limits are placed both on the [`RPCMsg`](https://github.com/libp2p/specs/bl Clients MUST reject and MUST NOT emit or propagate messages whose size exceed the following limits: -* the size of the encoded `RPCMsg`, including control messages and framing, must not exceed `max_message_size()` +* the size of the encoded `RPCMsg`, including control messages, framing, topics etc, must not exceed `max_message_size()` * the size of the compressed payload in the `Message.data` field must not exceed `max_compressed_len(MAX_PAYLOAD_SIZE)`. * the size of the uncompressed payload must not exceed `MAX_PAYLOAD_SIZE` or the [type-specific SSZ bound](#what-are-ssz-type-size-bounds), whichever is lower. From 305f30e89505615d924e42f81105b6de104c8e74 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 10:15:13 -0600 Subject: [PATCH 04/14] Bump circleci's cached venv key --- .circleci/config.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 1de55179d4..37e094e1de 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -35,13 +35,13 @@ commands: description: "Restore the cache with pyspec keys" steps: - restore_cached_venv: - venv_name: v30-pyspec + venv_name: v31-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} save_pyspec_cached_venv: description: Save a venv into a cache with pyspec keys" steps: - save_cached_venv: - venv_name: v30-pyspec + venv_name: v31-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} venv_path: ./venv jobs: From 702722fe6995d2917aa9d6e1eb23085dde539b06 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 10:25:21 -0600 Subject: [PATCH 05/14] Bump circleci's cached repo key --- .circleci/config.yml | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 37e094e1de..9be3106db1 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -53,16 +53,16 @@ jobs: # Restore git repo at point close to target branch/revision, to speed up checkout - restore_cache: keys: - - v3-specs-repo-{{ .Branch }}-{{ .Revision }} - - v3-specs-repo-{{ .Branch }}- - - v3-specs-repo- + - v4-specs-repo-{{ .Branch }}-{{ .Revision }} + - v4-specs-repo-{{ .Branch }}- + - v4-specs-repo- - checkout - run: name: Clean up git repo to reduce cache size command: git gc # Save the git checkout as a cache, to make cloning next time faster. - save_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} paths: - ~/specs-repo install_pyspec_test: @@ -71,7 +71,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Install pyspec requirements @@ -83,7 +83,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -96,7 +96,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -109,7 +109,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -122,7 +122,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -135,7 +135,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -148,7 +148,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -161,7 +161,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -174,7 +174,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -187,7 +187,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Install doctoc From b1205ef967de705957df1f50e6c5453d8bde09de Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 10:42:46 -0600 Subject: [PATCH 06/14] Revert "Bump circleci's cached venv key" This reverts commit 305f30e89505615d924e42f81105b6de104c8e74. --- .circleci/config.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 9be3106db1..d142e4ac24 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -35,13 +35,13 @@ commands: description: "Restore the cache with pyspec keys" steps: - restore_cached_venv: - venv_name: v31-pyspec + venv_name: v30-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} save_pyspec_cached_venv: description: Save a venv into a cache with pyspec keys" steps: - save_cached_venv: - venv_name: v31-pyspec + venv_name: v30-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} venv_path: ./venv jobs: From 46f1dde2b7fd487b107a69b90aeb60366da762cf Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 10:42:56 -0600 Subject: [PATCH 07/14] Revert "Bump circleci's cached repo key" This reverts commit 702722fe6995d2917aa9d6e1eb23085dde539b06. --- .circleci/config.yml | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index d142e4ac24..1de55179d4 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -53,16 +53,16 @@ jobs: # Restore git repo at point close to target branch/revision, to speed up checkout - restore_cache: keys: - - v4-specs-repo-{{ .Branch }}-{{ .Revision }} - - v4-specs-repo-{{ .Branch }}- - - v4-specs-repo- + - v3-specs-repo-{{ .Branch }}-{{ .Revision }} + - v3-specs-repo-{{ .Branch }}- + - v3-specs-repo- - checkout - run: name: Clean up git repo to reduce cache size command: git gc # Save the git checkout as a cache, to make cloning next time faster. - save_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} paths: - ~/specs-repo install_pyspec_test: @@ -71,7 +71,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Install pyspec requirements @@ -83,7 +83,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -96,7 +96,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -109,7 +109,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -122,7 +122,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -135,7 +135,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -148,7 +148,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -161,7 +161,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -174,7 +174,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Run py-tests @@ -187,7 +187,7 @@ jobs: working_directory: ~/specs-repo steps: - restore_cache: - key: v4-specs-repo-{{ .Branch }}-{{ .Revision }} + key: v3-specs-repo-{{ .Branch }}-{{ .Revision }} - restore_pyspec_cached_venv - run: name: Install doctoc From cb4ed99f4e889c754dba3f2aadad3ed744c00e23 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 10:54:15 -0600 Subject: [PATCH 08/14] Fix linting errors for new functions --- specs/phase0/p2p-interface.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index 8bc05844c1..e400dff58c 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -17,6 +17,8 @@ - [Configuration](#configuration) - [MetaData](#metadata) - [Maximum message sizes](#maximum-message-sizes) + - [`max_compressed_len`](#max_compressed_len) + - [`max_message_size`](#max_message_size) - [The gossip domain: gossipsub](#the-gossip-domain-gossipsub) - [Topics and messages](#topics-and-messages) - [Global topics](#global-topics) @@ -236,15 +238,21 @@ and will in most cases be out of sync with the ENR sequence number. Maximum message sizes are derived from the maximum payload size that the network can carry according to the following functions: +#### `max_compressed_len` + +```python +def max_compressed_len(n: uint64) -> uint64: + # Worst-case compressed length for a given payload of size n when using snappy + # https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47 + return uint64(32 + n + n / 6) +``` + +#### `max_message_size` + ```python -def max_compressed_len(n): - # Worst-case compressed length for a given payload of size n when using snappy - # https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47 - return int(32 + n + n / 6) - -def max_message_size(): - # Allow 1024 bytes for framing and encoding overhead but at least 1MB in case MAX_PAYLOAD_SIZE is small. - return max(max_compressed_len(MAX_PAYLOAD_SIZE) + 1024, 1024*1024) +def max_message_size() -> uint64: + # Allow 1024 bytes for framing and encoding overhead but at least 1MiB in case MAX_PAYLOAD_SIZE is small. + return max(max_compressed_len(MAX_PAYLOAD_SIZE) + 1024, 1024 * 1024) ``` ### The gossip domain: gossipsub From d41b7bddf5e77c8a7d49832b11485f53f7c5e83f Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Tue, 17 Dec 2024 11:35:49 -0600 Subject: [PATCH 09/14] Bump venv cache key again --- .circleci/config.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 1de55179d4..38bd6f422d 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -35,13 +35,13 @@ commands: description: "Restore the cache with pyspec keys" steps: - restore_cached_venv: - venv_name: v30-pyspec + venv_name: v32-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} save_pyspec_cached_venv: description: Save a venv into a cache with pyspec keys" steps: - save_cached_venv: - venv_name: v30-pyspec + venv_name: v32-pyspec reqs_checksum: cache-{{ checksum "setup.py" }}-{{ checksum "requirements_preinstallation.txt" }} venv_path: ./venv jobs: From 44cecd2caa0345bae46341641738608f1d8e58fe Mon Sep 17 00:00:00 2001 From: Jacek Sieka Date: Tue, 7 Jan 2025 18:31:04 +0100 Subject: [PATCH 10/14] fix bellatrix constant too --- specs/bellatrix/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/bellatrix/p2p-interface.md b/specs/bellatrix/p2p-interface.md index 1f4c815660..5d8425e888 100644 --- a/specs/bellatrix/p2p-interface.md +++ b/specs/bellatrix/p2p-interface.md @@ -148,8 +148,8 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: #### Why was the max gossip message size increased at Bellatrix? With the addition of `ExecutionPayload` to `BeaconBlock`s, there is a dynamic -field -- `transactions` -- which can validly exceed the `GOSSIP_MAX_SIZE` limit (1 MiB) put in -place at Phase 0, so GOSSIP_MAX_SIZE has increased to 10 Mib on the network. +field -- `transactions` -- which can validly exceed the `MAX_PAYLOAD_SIZE` limit (1 MiB) put in +place at Phase 0, so MAX_PAYLOAD_SIZE has increased to 10 Mib on the network. At the `GAS_LIMIT` (~30M) currently seen on mainnet in 2021, a single transaction filled entirely with data at a cost of 16 gas per byte can create a valid `ExecutionPayload` of ~2 MiB. Thus we need a size limit to at least account for From 454bd57cd0fd0ead7012b1ab81460f2fd7a5f49f Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Wed, 8 Jan 2025 14:20:01 -0600 Subject: [PATCH 11/14] Update config files & fix some nits --- configs/mainnet.yaml | 4 +--- configs/minimal.yaml | 4 +--- specs/bellatrix/p2p-interface.md | 2 +- specs/phase0/p2p-interface.md | 2 +- 4 files changed, 4 insertions(+), 8 deletions(-) diff --git a/configs/mainnet.yaml b/configs/mainnet.yaml index deb3dcf5fe..e54db49661 100644 --- a/configs/mainnet.yaml +++ b/configs/mainnet.yaml @@ -115,15 +115,13 @@ DEPOSIT_CONTRACT_ADDRESS: 0x00000000219ab540356cBB839Cbe05303d7705Fa # Networking # --------------------------------------------------------------- # `10 * 2**20` (= 10485760, 10 MiB) -GOSSIP_MAX_SIZE: 10485760 +MAX_PAYLOAD_SIZE: 10485760 # `2**10` (= 1024) MAX_REQUEST_BLOCKS: 1024 # `2**8` (= 256) EPOCHS_PER_SUBNET_SUBSCRIPTION: 256 # `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) MIN_EPOCHS_FOR_BLOCK_REQUESTS: 33024 -# `10 * 2**20` (=10485760, 10 MiB) -MAX_CHUNK_SIZE: 10485760 # 5s TTFB_TIMEOUT: 5 # 10s diff --git a/configs/minimal.yaml b/configs/minimal.yaml index 460474ebf7..a15314bb1f 100644 --- a/configs/minimal.yaml +++ b/configs/minimal.yaml @@ -116,15 +116,13 @@ DEPOSIT_CONTRACT_ADDRESS: 0x1234567890123456789012345678901234567890 # Networking # --------------------------------------------------------------- # `10 * 2**20` (= 10485760, 10 MiB) -GOSSIP_MAX_SIZE: 10485760 +MAX_PAYLOAD_SIZE: 10485760 # `2**10` (= 1024) MAX_REQUEST_BLOCKS: 1024 # `2**8` (= 256) EPOCHS_PER_SUBNET_SUBSCRIPTION: 256 # [customized] `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 272) MIN_EPOCHS_FOR_BLOCK_REQUESTS: 272 -# `10 * 2**20` (=10485760, 10 MiB) -MAX_CHUNK_SIZE: 10485760 # 5s TTFB_TIMEOUT: 5 # 10s diff --git a/specs/bellatrix/p2p-interface.md b/specs/bellatrix/p2p-interface.md index 5d8425e888..b2d28cf1f4 100644 --- a/specs/bellatrix/p2p-interface.md +++ b/specs/bellatrix/p2p-interface.md @@ -149,7 +149,7 @@ Per `context = compute_fork_digest(fork_version, genesis_validators_root)`: With the addition of `ExecutionPayload` to `BeaconBlock`s, there is a dynamic field -- `transactions` -- which can validly exceed the `MAX_PAYLOAD_SIZE` limit (1 MiB) put in -place at Phase 0, so MAX_PAYLOAD_SIZE has increased to 10 Mib on the network. +place at Phase 0, so MAX_PAYLOAD_SIZE has increased to 10 MiB on the network. At the `GAS_LIMIT` (~30M) currently seen on mainnet in 2021, a single transaction filled entirely with data at a cost of 16 gas per byte can create a valid `ExecutionPayload` of ~2 MiB. Thus we need a size limit to at least account for diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index e400dff58c..f3d9038abd 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -199,7 +199,7 @@ This section outlines configurations that are used in this spec. | Name | Value | Description | |---|---|---| -| `MAX_PAYLOAD_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed payload in gossipsub messages / RPC chunks. | +| `MAX_PAYLOAD_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed payload in gossipsub messages / RPC chunks | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | | `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on a subnet subscription (~27 hours) | | `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | From 5127929733ed14c5f06b0dc675f575daaac9a155 Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Wed, 8 Jan 2025 14:43:54 -0600 Subject: [PATCH 12/14] Try to polish new paragraphs a bit --- specs/phase0/p2p-interface.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index f3d9038abd..ea51d96dfd 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -242,7 +242,7 @@ Maximum message sizes are derived from the maximum payload size that the network ```python def max_compressed_len(n: uint64) -> uint64: - # Worst-case compressed length for a given payload of size n when using snappy + # Worst-case compressed length for a given payload of size n when using snappy: # https://github.com/google/snappy/blob/32ded457c0b1fe78ceb8397632c416568d6714a0/snappy.cc#L218C1-L218C47 return uint64(32 + n + n / 6) ``` @@ -534,9 +534,9 @@ Size limits are placed both on the [`RPCMsg`](https://github.com/libp2p/specs/bl Clients MUST reject and MUST NOT emit or propagate messages whose size exceed the following limits: -* the size of the encoded `RPCMsg`, including control messages, framing, topics etc, must not exceed `max_message_size()` -* the size of the compressed payload in the `Message.data` field must not exceed `max_compressed_len(MAX_PAYLOAD_SIZE)`. -* the size of the uncompressed payload must not exceed `MAX_PAYLOAD_SIZE` or the [type-specific SSZ bound](#what-are-ssz-type-size-bounds), whichever is lower. +* The size of the encoded `RPCMsg` (including control messages, framing, topics, etc) must not exceed `max_message_size()`. +* The size of the compressed payload in the `Message.data` field must not exceed `max_compressed_len(MAX_PAYLOAD_SIZE)`. +* The size of the uncompressed payload must not exceed `MAX_PAYLOAD_SIZE` or the [type-specific SSZ bound](#what-are-ssz-type-size-bounds), whichever is lower. ### The Req/Resp domain @@ -1715,19 +1715,17 @@ It is advisable to derive these lengths from the SSZ type definitions in use, to #### Why is the message size defined in terms of application payload? -When transmitting messages over gossipsub and / or req/resp, we want to ensure that the same payload sizes are supported no matter the underlying transport, decoupling the consensus layer from libp2p-induced overhead and the particular transmission strategy. +When transmitting messages over gossipsub and/or the req/resp domain, we want to ensure that the same payload sizes are supported regardless of the underlying transport, decoupling the consensus layer from libp2p-induced overhead and the particular transmission strategy. -To derive "encoded size limits" from desired application sizes we take into account snappy compression and framing overhead. +To derive "encoded size limits" from desired application sizes, we take into account snappy compression and framing overhead. -In the case of gossipsub, the protocol supports sending multiple application payloads as well as mixing application data with control messages in each gossipsub frame - the limit is set such that at least one max-sized application-level message together with a small amount (1kb) of gossipsub overhead is allowed - implementations are free to pack multiple smaller application messages into a single gossipsub frame, and / or combine it with control messages as they see fit. - -The limit is set on the uncompressed payload size in particular to protect against decompression bombs - although +In the case of gossipsub, the protocol supports sending multiple application payloads as well as mixing application data with control messages in each gossipsub frame. The limit is set such that at least one max-sized application-level message together with a small amount (1 KiB) of gossipsub overhead is allowed. Implementations are free to pack multiple smaller application messages into a single gossipsub frame, and/or combine it with control messages as they see fit. #### Why is there a limit on message sizes at all? -The message size limit protects against several forms of DoS and network-based amplification attacks and provide upper bounds for resource (network, memory) usage in the client based on protocol requirements to decode, buffer, cache, store and re-transmit messages which in turn translate into performance and protection tradeoffs, ensuring capacity to handle worst cases during recovery from network instability. +The message size limit protects against several forms of DoS and network-based amplification attacks and provides upper bounds for resource (network, memory) usage in the client based on protocol requirements to decode, buffer, cache, store and re-transmit messages which in turn translate into performance and protection tradeoffs, ensuring capacity to handle worst cases during recovery from network instability. -In particular, blocks which at the time of writing is the only message type without a practical SSZ-derived upper bound on size cannot be fully verified synchronously as part of gossipsub validity checks meaning that there exist cases where invalid messages signed by a validator may be amplified by the network. +In particular, blocks—-currently the only message type without a practical SSZ-derived upper bound on size—-cannot be fully verified synchronously as part of gossipsub validity checks. This means that there exist cases where invalid messages signed by a validator may be amplified by the network. ## libp2p implementations matrix From e8eb367da26e908d0f0e7219bc9c5ad45e5b5e7e Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Wed, 8 Jan 2025 14:46:16 -0600 Subject: [PATCH 13/14] Fix two more small nits --- specs/phase0/p2p-interface.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index ea51d96dfd..ab3306d235 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -199,7 +199,7 @@ This section outlines configurations that are used in this spec. | Name | Value | Description | |---|---|---| -| `MAX_PAYLOAD_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed payload in gossipsub messages / RPC chunks | +| `MAX_PAYLOAD_SIZE` | `10 * 2**20` (= 10485760, 10 MiB) | The maximum allowed size of uncompressed payload in gossipsub messages and RPC chunks | | `MAX_REQUEST_BLOCKS` | `2**10` (= 1024) | Maximum number of blocks in a single request | | `EPOCHS_PER_SUBNET_SUBSCRIPTION` | `2**8` (= 256) | Number of epochs on a subnet subscription (~27 hours) | | `MIN_EPOCHS_FOR_BLOCK_REQUESTS` | `MIN_VALIDATOR_WITHDRAWABILITY_DELAY + CHURN_LIMIT_QUOTIENT // 2` (= 33024, ~5 months) | The minimum epoch range over which a node must serve blocks | @@ -294,7 +294,7 @@ This defines both the type of data being sent on the topic and how the data fiel - `Encoding` - the encoding strategy describes a specific representation of bytes that will be transmitted over the wire. See the [Encodings](#Encodings) section for further details. -Clients MUST reject messages with unknown topic. +Clients MUST reject messages with an unknown topic. *Note*: `ForkDigestValue` is composed of values that are not known until the genesis block/state are available. Due to this, clients SHOULD NOT subscribe to gossipsub topics until these genesis values are known. From d867b84f093fe5270da7e7a49a9c9ea1be7c538c Mon Sep 17 00:00:00 2001 From: Justin Traglia Date: Wed, 8 Jan 2025 15:34:06 -0600 Subject: [PATCH 14/14] Add back remark about compression bombs --- specs/phase0/p2p-interface.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/specs/phase0/p2p-interface.md b/specs/phase0/p2p-interface.md index ab3306d235..1196fca90d 100644 --- a/specs/phase0/p2p-interface.md +++ b/specs/phase0/p2p-interface.md @@ -1721,6 +1721,8 @@ To derive "encoded size limits" from desired application sizes, we take into acc In the case of gossipsub, the protocol supports sending multiple application payloads as well as mixing application data with control messages in each gossipsub frame. The limit is set such that at least one max-sized application-level message together with a small amount (1 KiB) of gossipsub overhead is allowed. Implementations are free to pack multiple smaller application messages into a single gossipsub frame, and/or combine it with control messages as they see fit. +The limit is set on the uncompressed payload size in particular to protect against decompression bombs. + #### Why is there a limit on message sizes at all? The message size limit protects against several forms of DoS and network-based amplification attacks and provides upper bounds for resource (network, memory) usage in the client based on protocol requirements to decode, buffer, cache, store and re-transmit messages which in turn translate into performance and protection tradeoffs, ensuring capacity to handle worst cases during recovery from network instability.