From 201c73e22e509c859c5a82dab0d5a63008844f3d Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 19 May 2021 00:02:25 +0200 Subject: [PATCH 01/21] adr-40: use prefix store instead of multistore --- ...r-040-storage-and-smt-state-commitments.md | 37 +++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 0f009e0b4cf0..3f5650dbbe6d 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -119,6 +119,41 @@ We need to be able to process transactions and roll-back state updates if a tran We identified use-cases, where modules will need to save an object commitment without storing an object itself. Sometimes clients are receiving complex objects, and they have no way to prove a correctness of that object without knowing the storage layout. For those use cases it would be easier to commit to the object without storing it directly. +### Remove MultiStore + +IAVL based store adds additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using it's own instance of IAVL, but in the current implementation, all instances share the same database. +The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). + +We propose to remove the MultiStore from the SDK, and use a single instance of `SC`. To improve usability, we should extend the `KVStore` interface with _prefix store_: + +``` +type KVStore interface { + ... current KVStore + + WithPrefix(prefix string) KVStore +} +``` + +The `WithPrefix` method will create a proxy object for the parent object and will prepend `prefix` to a key parameter of all key-value operations. +The following invariant should be applied + +``` +for each OP in [Get Has, Set, ...] + store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) +``` + +#### Optimization: use compression for prefix keys + +Moreover we can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes. And for best results it will need an frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding the above invariant should have the following shape: + +``` +for each OP in [Get Has, Set, ...] + store.WithPrefix(prefix).OP(key) == store.OP(store.Code(prefix) + key) +``` + +Where `store.Code(prefix)` is a Huffman Code of `prefix` in the given `store`. + + ## Consequences @@ -133,6 +168,8 @@ We change the storage layout of the state machine, a storage hard fork and netwo + Decoupling state from state commitment introduce better engineering opportunities for further optimizations and better storage patterns. + Performance improvements. + Joining SMT based camp which has wider and proven adoption than IAVL. Example projects which decided on SMT: Ethereum2, Diem (Libra), Trillan, Tezos, LazyLedger. ++ Multistore removal fixes a long standing issues with current MultiStore design. + ### Negative From c432f5ed487ee96d0b623594539440cc6d272d4c Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 19 May 2021 00:55:07 +0200 Subject: [PATCH 02/21] add note about prefix.Store --- docs/architecture/adr-040-storage-and-smt-state-commitments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 3f5650dbbe6d..7ef4baa175e6 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -134,7 +134,7 @@ type KVStore interface { } ``` -The `WithPrefix` method will create a proxy object for the parent object and will prepend `prefix` to a key parameter of all key-value operations. +The `WithPrefix` method will create a proxy object for the parent object and will prepend `prefix` to a key parameter of all key-value operations (similar to the `store/prefix.Store` implementation). The following invariant should be applied ``` From bbc692f79cd6bd7be673cfe283741d12407acb9d Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Fri, 28 May 2021 15:35:53 +0200 Subject: [PATCH 03/21] Update SC and SS setup information and historical versions sepc --- ...adr-040-storage-and-smt-state-commitments.md | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index ac846387632a..285e0bb7c115 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -48,7 +48,7 @@ For data access we propose 2 additional KV buckets (namespaces for the key-value 2. B2: `hash(key, value) → key`: a reverse index to get a key from an SMT path. Recall that SMT will store `(k, v)` as `(hash(k), hash(key, value))`. So, we can get an object value by composing `SMT_path → B2 → B1`. 3. we could use more buckets to optimize the app usage if needed. -Above, we propose to use a KV DB. However, for the state machine, we could use an RDBMS, which we discuss below. +We propose to use a KV database for both `SS` and `SC` - each run by it's own database instance. This design allows to separate the `SS` and `SC` in different hardware units and support for more complex setup scenarios and to improve DB performance (essentially this will create 2 store shards: one for `SS` and another for `SC`). Moreover we will be able to configure database for `SS` and `SC` separately. ### Requirements @@ -64,6 +64,7 @@ State Commitment requirements: + fast updates + tree path should be short ++ query historical commitment proofs using ICS-23 standard. + pruning (garbage collection) ### LazyLedger SMT for State Commitment @@ -74,27 +75,27 @@ A Sparse Merkle tree is based on the idea of a complete Merkle tree of an intrac Below, with simple _snapshot_ we refer to a database snapshot mechanism, not to a _ABCI snapshot sync_. The latter will be referred as _snapshot sync_ (which will directly use DB snapshot as described below). -Database snapshot is a view of DB state at a certain time or transaction. It's not a full copy of a database (it would be too big), usually a snapshot mechanism is based on a _copy on write_ and it allows to efficiently deliver DB state at a certain stage. -Some DB engines support snapshotting. Hence, we propose to reuse that functionality for the state sync and versioning (described below). It will the supported DB engines to ones which efficiently implement snapshots. In a final section we will discuss evaluated DBs. +Database snapshot is a view of DB state at a certain time or transaction. It's not a full copy of a database (it would be too big). Usually a snapshot mechanism is based on a _copy on write_ and it allows to efficiently deliver DB state at a certain stage. +Some DB engines support snapshotting. Hence, we propose to reuse that functionality for the state sync and versioning (described below). We limit the supported DB engines to ones which efficiently implement snapshots. In a final section we discuss the evaluated DBs. One of the Stargate core features is a _snapshot sync_ delivered in the `/snapshot` package. It provides a way to trustlessly sync a blockchain without repeating all transactions from the genesis. This feature is implemented in SDK and requires storage support. Currently IAVL is the only supported backend. It works by streaming to a client a snapshot of a `SS` at a certain version together with a header chain. -A new `SS` snapshot will be created in every `EndBlocker` and identified by a block height. The `rootmulti.Store` keeps track of the available snapshots to offer `SS` at a certain version. The `rootmulti.Store` implements the `CommitMultiStore` interface, which encapsulates a `Committer` interface. `Committer` has a `Commit`, `SetPruning`, `GetPruning` functions which will be used for creating and removing snapshots. The `rootStore.Commit` function creates a new snapshot and increments the version on each call, and checks if it needs to remove old versions. We will need to update the SMT interface to implement the `Committer` interface. +A new database snapshot will be created in every `EndBlocker` and identified by a block height. The `rootmulti.Store` keeps track of the available snapshots to offer `SS` at a certain version. The `rootmulti.Store` implements the `CommitMultiStore` interface, which encapsulates a `Committer` interface. `Committer` has a `Commit`, `SetPruning`, `GetPruning` functions which will be used for creating and removing snapshots. The `rootStore.Commit` function creates a new snapshot and increments the version on each call, and checks if it needs to remove old versions. We will need to update the SMT interface to implement the `Committer` interface. NOTE: `Commit` must be called exactly once per block. Otherwise we risk going out of sync for the version number and block height. NOTE: For the SDK storage, we may consider splitting that interface into `Committer` and `PruningCommitter` - only the multiroot should implement `PruningCommitter` (cache and prefix store don't need pruning). Number of historical versions for `abci.RequestQuery` and state sync snapshots is part of a node configuration, not a chain configuration (configuration implied by the blockchain consensus). A configuration should allow to specify number of past blocks and number of past blocks modulo some number (eg: 100 past blocks and one snapshot every 100 blocks for past 2000 blocks). Archival nodes can keep all past versions. -Pruning old snapshots is effectively done by a database. Whenever we update a record in `SC`, SMT won't update nodes - instead it creates new nodes on the update path, without removing the old one. Since we are snapshoting each block, we need to update that mechanism to immediately remove orphaned nodes from the storage. This is a safe operation - snapshots will keep track of the records which should be available for past versions. +Pruning old snapshots is effectively done by a database. Whenever we update a record in `SC`, SMT won't update nodes - instead it creates new nodes on the update path, without removing the old one. Since we are snapshoting each block, we need to change that mechanism to immediately remove orphaned nodes from the database. This is a safe operation - snapshots will keep track of the records and make it available when accessing past versions. -To manage the active snapshots we will either us a DB _max number of snapshots_ option (if available), or will remove snapshots in the `EndBlocker`. The latter option can be done efficiently by identifying snapshots with block height. +To manage the active snapshots we will either us a DB _max number of snapshots_ option (if available), or will remove DB snapshots in the `EndBlocker`. The latter option can be done efficiently by identifying snapshots with block height and calling a store function to remove past versions. #### Accessing old state versions One of the functional requirements is to access old state. This is done through `abci.RequestQuery` structure. The version is specified by a block height (so we query for an object by a key `K` at block height `H`). The number of old versions supported for `abci.RequestQuery` is configurable. Accessing an old state is done by using available snapshots. -`abci.RequestQuery` doesn't need old state of `SC`. So, for efficiency, we should keep `SC` and `SS` in different databases (however using the same DB engine). +`abci.RequestQuery` doesn't need old state of `SC` unless the `prove=true` parameter is set. The SMT merkle proof must be included in the `abci.ResponseQuery` only if both `SC` and `SS` have a snapshot for requested version. -Moreover, SDK could provide a way to directly access the state. However, a state machine shouldn't do that - since the number of snapshots is configurable, it would lead to nondeterministic execution. +Moreover, SDK could provide a way to directly access a historical state. However, a state machine shouldn't do that - since the number of snapshots is configurable, it would lead to nondeterministic execution. We positively [validated](https://github.com/cosmos/cosmos-sdk/discussions/8297) a versioning and snapshot mechanism for querying old state with regards to the database we evaluated. From 0612088cfa2ad1534fa41ba0bcc97eedf85b0252 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Fri, 28 May 2021 16:04:09 +0200 Subject: [PATCH 04/21] add note about key prefix optimization --- .../adr-040-storage-and-smt-state-commitments.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 285e0bb7c115..56fd7e9b49a7 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -117,7 +117,7 @@ We identified use-cases, where modules will need to save an object commitment wi IAVL based store adds additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using it's own instance of IAVL, but in the current implementation, all instances share the same database. The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). -We propose to remove the MultiStore from the SDK, and use a single instance of `SC`. To improve usability, we should extend the `KVStore` interface with _prefix store_: +We propose to remove the MultiStore from the SDK, and use a single instance of `SC`. To improve usability, we should extend the `KVStore` interface with _prefix store_. Each module reserves a key address space using a Module Store Key. This key is prefixed to all KV operations. ``` type KVStore interface { @@ -135,9 +135,9 @@ for each OP in [Get Has, Set, ...] store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) ``` -#### Optimization: use compression for prefix keys +### Optimization: compress module keys -Moreover we can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes. And for best results it will need an frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding the above invariant should have the following shape: +We can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes (module store keys) a priori. And for best results it will need an frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding the above invariant should have the following form: ``` for each OP in [Get Has, Set, ...] @@ -146,6 +146,14 @@ for each OP in [Get Has, Set, ...] Where `store.Code(prefix)` is a Huffman Code of `prefix` in the given `store`. +To avoid conflicts in the address space, we need to assure that in the set of prefix codes, there are not two elements where one is a prefix of another: + +``` +for each k1, k2 \in {store.Code(p): p \in StoreModuleKeys} + assert( !k1.hasPrefix(k2) ) +``` + +NOTE: We need to assure that the codes won't change. Huffman Coding depends on the keys and it's frequencies - so we would need to generate the codes and then fix the mapping in a static variable. ## Consequences From c12c5ec2c05687f2adb726d6f9e575a7b243d9dc Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Thu, 3 Jun 2021 02:27:42 +0200 Subject: [PATCH 05/21] rephrased the changes related to multistore --- .../adr-040-storage-and-smt-state-commitments.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 56fd7e9b49a7..3653ea625a24 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -114,10 +114,12 @@ We identified use-cases, where modules will need to save an object commitment wi ### Remove MultiStore -IAVL based store adds additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using it's own instance of IAVL, but in the current implementation, all instances share the same database. +IAVL based store adds additional layer in the SDK store construction - the `MultiStore` structure. The multi store exists to support the modularity of the Cosmos SDK - each module is using it's own instance of IAVL, but in the current implementation, all instances share the same database. The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). -We propose to remove the MultiStore from the SDK, and use a single instance of `SC`. To improve usability, we should extend the `KVStore` interface with _prefix store_. Each module reserves a key address space using a Module Store Key. This key is prefixed to all KV operations. +We propose to remove the MultiStore concept from the SDK, and use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusions, we should rename `MultiStore` interface to `RootStore` interface and make sure that the `rootStore` object will implement it. + +Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: ``` type KVStore interface { From 60af1b01d82f7422806d0f3553057eb2ff61c070 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Fri, 4 Jun 2021 12:19:17 +0200 Subject: [PATCH 06/21] Apply suggestions from code review Co-authored-by: Ryan Christoffersen <12519942+ryanchristo@users.noreply.github.com> --- ...r-040-storage-and-smt-state-commitments.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 3653ea625a24..721cbefdee91 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -48,7 +48,7 @@ For data access we propose 2 additional KV buckets (namespaces for the key-value 2. B2: `hash(key, value) → key`: a reverse index to get a key from an SMT path. Recall that SMT will store `(k, v)` as `(hash(k), hash(key, value))`. So, we can get an object value by composing `SMT_path → B2 → B1`. 3. we could use more buckets to optimize the app usage if needed. -We propose to use a KV database for both `SS` and `SC` - each run by it's own database instance. This design allows to separate the `SS` and `SC` in different hardware units and support for more complex setup scenarios and to improve DB performance (essentially this will create 2 store shards: one for `SS` and another for `SC`). Moreover we will be able to configure database for `SS` and `SC` separately. +We propose to use a KV database for both `SS` and `SC` - each run by its own database instance. This design allows for the separation of `SS` and `SC` into different hardware units, providing support for more complex setup scenarios and improving DB performance (essentially this will create two store shards: one for `SS` and another for `SC`). Moreover, we will be able to configure databases for `SS` and `SC` separately. ### Requirements @@ -64,7 +64,7 @@ State Commitment requirements: + fast updates + tree path should be short -+ query historical commitment proofs using ICS-23 standard. ++ query historical commitment proofs using ICS-23 standard + pruning (garbage collection) ### LazyLedger SMT for State Commitment @@ -75,7 +75,7 @@ A Sparse Merkle tree is based on the idea of a complete Merkle tree of an intrac Below, with simple _snapshot_ we refer to a database snapshot mechanism, not to a _ABCI snapshot sync_. The latter will be referred as _snapshot sync_ (which will directly use DB snapshot as described below). -Database snapshot is a view of DB state at a certain time or transaction. It's not a full copy of a database (it would be too big). Usually a snapshot mechanism is based on a _copy on write_ and it allows to efficiently deliver DB state at a certain stage. +Database snapshot is a view of DB state at a certain time or transaction. It's not a full copy of a database (it would be too big). Usually a snapshot mechanism is based on a _copy on write_ and it allows DB state to be efficiently delivered at a certain stage. Some DB engines support snapshotting. Hence, we propose to reuse that functionality for the state sync and versioning (described below). We limit the supported DB engines to ones which efficiently implement snapshots. In a final section we discuss the evaluated DBs. One of the Stargate core features is a _snapshot sync_ delivered in the `/snapshot` package. It provides a way to trustlessly sync a blockchain without repeating all transactions from the genesis. This feature is implemented in SDK and requires storage support. Currently IAVL is the only supported backend. It works by streaming to a client a snapshot of a `SS` at a certain version together with a header chain. @@ -86,16 +86,16 @@ NOTE: For the SDK storage, we may consider splitting that interface into `Commit Number of historical versions for `abci.RequestQuery` and state sync snapshots is part of a node configuration, not a chain configuration (configuration implied by the blockchain consensus). A configuration should allow to specify number of past blocks and number of past blocks modulo some number (eg: 100 past blocks and one snapshot every 100 blocks for past 2000 blocks). Archival nodes can keep all past versions. -Pruning old snapshots is effectively done by a database. Whenever we update a record in `SC`, SMT won't update nodes - instead it creates new nodes on the update path, without removing the old one. Since we are snapshoting each block, we need to change that mechanism to immediately remove orphaned nodes from the database. This is a safe operation - snapshots will keep track of the records and make it available when accessing past versions. +Pruning old snapshots is effectively done by a database. Whenever we update a record in `SC`, SMT won't update nodes - instead it creates new nodes on the update path, without removing the old one. Since we are snapshotting each block, we need to change that mechanism to immediately remove orphaned nodes from the database. This is a safe operation - snapshots will keep track of the records and make it available when accessing past versions. -To manage the active snapshots we will either us a DB _max number of snapshots_ option (if available), or will remove DB snapshots in the `EndBlocker`. The latter option can be done efficiently by identifying snapshots with block height and calling a store function to remove past versions. +To manage the active snapshots we will either use a DB _max number of snapshots_ option (if available), or we will remove DB snapshots in the `EndBlocker`. The latter option can be done efficiently by identifying snapshots with block height and calling a store function to remove past versions. #### Accessing old state versions One of the functional requirements is to access old state. This is done through `abci.RequestQuery` structure. The version is specified by a block height (so we query for an object by a key `K` at block height `H`). The number of old versions supported for `abci.RequestQuery` is configurable. Accessing an old state is done by using available snapshots. `abci.RequestQuery` doesn't need old state of `SC` unless the `prove=true` parameter is set. The SMT merkle proof must be included in the `abci.ResponseQuery` only if both `SC` and `SS` have a snapshot for requested version. -Moreover, SDK could provide a way to directly access a historical state. However, a state machine shouldn't do that - since the number of snapshots is configurable, it would lead to nondeterministic execution. +Moreover, the SDK could provide a way to directly access a historical state. However, a state machine shouldn't do that - since the number of snapshots is configurable, it would lead to nondeterministic execution. We positively [validated](https://github.com/cosmos/cosmos-sdk/discussions/8297) a versioning and snapshot mechanism for querying old state with regards to the database we evaluated. @@ -114,10 +114,10 @@ We identified use-cases, where modules will need to save an object commitment wi ### Remove MultiStore -IAVL based store adds additional layer in the SDK store construction - the `MultiStore` structure. The multi store exists to support the modularity of the Cosmos SDK - each module is using it's own instance of IAVL, but in the current implementation, all instances share the same database. +IAVL based store adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). -We propose to remove the MultiStore concept from the SDK, and use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusions, we should rename `MultiStore` interface to `RootStore` interface and make sure that the `rootStore` object will implement it. +We propose to remove the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore` and make sure that the `rootStore` object implements `RootStore`. Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: @@ -139,7 +139,7 @@ for each OP in [Get Has, Set, ...] ### Optimization: compress module keys -We can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes (module store keys) a priori. And for best results it will need an frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding the above invariant should have the following form: +We can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes (module store keys) a priori. And for best results it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the above invariant should have the following form: ``` for each OP in [Get Has, Set, ...] @@ -155,7 +155,7 @@ for each k1, k2 \in {store.Code(p): p \in StoreModuleKeys} assert( !k1.hasPrefix(k2) ) ``` -NOTE: We need to assure that the codes won't change. Huffman Coding depends on the keys and it's frequencies - so we would need to generate the codes and then fix the mapping in a static variable. +NOTE: We need to assure that the codes won't change. Huffman Coding depends on the keys and its frequency - so we would need to generate the codes and then fix the mapping in a static variable. ## Consequences @@ -170,7 +170,7 @@ We change the storage layout of the state machine, a storage hard fork and netwo + Decoupling state from state commitment introduce better engineering opportunities for further optimizations and better storage patterns. + Performance improvements. + Joining SMT based camp which has wider and proven adoption than IAVL. Example projects which decided on SMT: Ethereum2, Diem (Libra), Trillan, Tezos, LazyLedger. -+ Multistore removal fixes a long standing issues with current MultiStore design. ++ Multistore removal fixes a longstanding issue with the current MultiStore design. ### Negative From 28bda90ea0f7792df107193d17deaffbd23f3b9c Mon Sep 17 00:00:00 2001 From: Federico Kunze <31522760+fedekunze@users.noreply.github.com> Date: Mon, 7 Jun 2021 03:06:02 -0400 Subject: [PATCH 07/21] Update docs/architecture/adr-040-storage-and-smt-state-commitments.md --- docs/architecture/adr-040-storage-and-smt-state-commitments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 721cbefdee91..3028894ceafb 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -139,7 +139,7 @@ for each OP in [Get Has, Set, ...] ### Optimization: compress module keys -We can consider a compression of prefix keys using Huffman Coding. It will require a knowledge of used prefixes (module store keys) a priori. And for best results it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the above invariant should have the following form: +We can consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the above invariant should have the following form: ``` for each OP in [Get Has, Set, ...] From aa1ded82ca3c129e9930fbde7f0ba406c996d548 Mon Sep 17 00:00:00 2001 From: Federico Kunze <31522760+fedekunze@users.noreply.github.com> Date: Mon, 7 Jun 2021 03:06:14 -0400 Subject: [PATCH 08/21] Update docs/architecture/adr-040-storage-and-smt-state-commitments.md --- docs/architecture/adr-040-storage-and-smt-state-commitments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 3028894ceafb..51ddd82d7d39 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -121,7 +121,7 @@ We propose to remove the multistore concept from the SDK, and to use a single in Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: -``` +```go type KVStore interface { ... current KVStore From 096526c0750b0ede1acb008fbe8aad0619b7274f Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 1 Sep 2021 15:08:13 +0200 Subject: [PATCH 09/21] Update docs/architecture/adr-040-storage-and-smt-state-commitments.md Co-authored-by: Aleksandr Bezobchuk --- docs/architecture/adr-040-storage-and-smt-state-commitments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 51ddd82d7d39..d0a29d35360a 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -115,6 +115,7 @@ We identified use-cases, where modules will need to save an object commitment wi ### Remove MultiStore IAVL based store adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. + The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). We propose to remove the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore` and make sure that the `rootStore` object implements `RootStore`. From ac7cde75ebcbc52941f8846e4cb5eefa1fb7dee5 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Tue, 14 Sep 2021 16:16:37 +0200 Subject: [PATCH 10/21] design update --- ...r-040-storage-and-smt-state-commitments.md | 45 +++++++++++++++---- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index d0a29d35360a..06654443aa47 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -80,7 +80,7 @@ Some DB engines support snapshotting. Hence, we propose to reuse that functional One of the Stargate core features is a _snapshot sync_ delivered in the `/snapshot` package. It provides a way to trustlessly sync a blockchain without repeating all transactions from the genesis. This feature is implemented in SDK and requires storage support. Currently IAVL is the only supported backend. It works by streaming to a client a snapshot of a `SS` at a certain version together with a header chain. -A new database snapshot will be created in every `EndBlocker` and identified by a block height. The `rootmulti.Store` keeps track of the available snapshots to offer `SS` at a certain version. The `rootmulti.Store` implements the `CommitMultiStore` interface, which encapsulates a `Committer` interface. `Committer` has a `Commit`, `SetPruning`, `GetPruning` functions which will be used for creating and removing snapshots. The `rootStore.Commit` function creates a new snapshot and increments the version on each call, and checks if it needs to remove old versions. We will need to update the SMT interface to implement the `Committer` interface. +A new database snapshot will be created in every `EndBlocker` and identified by a block height. The `root` store keeps track of the available snapshots to offer `SS` at a certain version. The `root` store implements the `RootStore` interface described below. In essence, `RootStore` encapsulates a `Committer` interface. `Committer` has a `Commit`, `SetPruning`, `GetPruning` functions which will be used for creating and removing snapshots. The `rootStore.Commit` function creates a new snapshot and increments the version on each call, and checks if it needs to remove old versions. We will need to update the SMT interface to implement the `Committer` interface. NOTE: `Commit` must be called exactly once per block. Otherwise we risk going out of sync for the version number and block height. NOTE: For the SDK storage, we may consider splitting that interface into `Committer` and `PruningCommitter` - only the multiroot should implement `PruningCommitter` (cache and prefix store don't need pruning). @@ -112,13 +112,13 @@ We need to be able to process transactions and roll-back state updates if a tran We identified use-cases, where modules will need to save an object commitment without storing an object itself. Sometimes clients are receiving complex objects, and they have no way to prove a correctness of that object without knowing the storage layout. For those use cases it would be easier to commit to the object without storing it directly. -### Remove MultiStore +### Reduce MultiStore -IAVL based store adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. +Stargate `/store` implementation (StoreV1) adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. -The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync problems (eg: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). +The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync (see: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). -We propose to remove the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore` and make sure that the `rootStore` object implements `RootStore`. +We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: @@ -131,16 +131,42 @@ type KVStore interface { ``` The `WithPrefix` method will create a proxy object for the parent object and will prepend `prefix` to a key parameter of all key-value operations (similar to the `store/prefix.Store` implementation). -The following invariant should be applied +The following invariant must hold: ``` for each OP in [Get Has, Set, ...] store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) ``` + +The `RootStore` will have the following interface: + +``` +TODO +``` + +In contrast to `MultiStore`, `RootStore` doesn't allow to mount dynamically sub stores. However it can inherit a multistore functionality in it's concrete implementation (which will be needed for IBC support). + +#### Merkle Proofs and IBC + +Currently IBC (v1.0) module merkle proof for a `(key, value)` consists of two elements `[storeKey, proof]`. Verification using 2 passes: +1. Checks that a `proof` is a valid merkle proof `(key, value)` using ICS-23 spec. +2. Then it checks that the `storeKey` and `root(proof)` hashes to the AppHash (App state commitment). + +Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Unfortunately, the straightforward implementation is breaking. Hence, we + +, it then verifies proof store key hashes to app_hash with simple merkle tree verification method. + + +RootStore merkle proofs, except for IBC module, will have only one pass. For module `M` and it's store key `S_m`, an object `O` with key `k` will be stored in `RootStore.WithPrefix(S_m)`. +This will create a record in `SC` at key `hash(S_m + key)`. + +However this breaks the existing IBC + + ### Optimization: compress module keys -We can consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the above invariant should have the following form: +We consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the `WithPrefix` invariant has the following form: ``` for each OP in [Get Has, Set, ...] @@ -149,7 +175,7 @@ for each OP in [Get Has, Set, ...] Where `store.Code(prefix)` is a Huffman Code of `prefix` in the given `store`. -To avoid conflicts in the address space, we need to assure that in the set of prefix codes, there are not two elements where one is a prefix of another: +To avoid conflicts in the address space, we need to assure that in the set of prefix codes there are no two elements where one is a prefix of another: ``` for each k1, k2 \in {store.Code(p): p \in StoreModuleKeys} @@ -172,12 +198,13 @@ We change the storage layout of the state machine, a storage hard fork and netwo + Performance improvements. + Joining SMT based camp which has wider and proven adoption than IAVL. Example projects which decided on SMT: Ethereum2, Diem (Libra), Trillan, Tezos, LazyLedger. + Multistore removal fixes a longstanding issue with the current MultiStore design. - ++ Simplifies merkle proofs - all modules, except IBC, have only one pass for merkle proof. ### Negative + Storage migration + LL SMT doesn't support pruning - we will need to add and test that functionality. ++ `SS` keys will have an overhead of a key prefix. This doesn't impact `SC` because all keys in `SC` have same size (they are hashed). ### Neutral From 4ee227e48cef4ea8c00eec35b53dbe2a99524551 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Thu, 16 Sep 2021 13:40:48 +0200 Subject: [PATCH 11/21] update merkle proofs --- .../adr-040-storage-and-smt-state-commitments.md | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 06654443aa47..6e632f5e8abf 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -153,16 +153,14 @@ Currently IBC (v1.0) module merkle proof for a `(key, value)` consists of two el 1. Checks that a `proof` is a valid merkle proof `(key, value)` using ICS-23 spec. 2. Then it checks that the `storeKey` and `root(proof)` hashes to the AppHash (App state commitment). -Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Unfortunately, the straightforward implementation is breaking. Hence, we +Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Unfortunately, the straightforward implementation is breaking because all keys in `SC` are hashed. So we can't simply make the last step. Even if we set the `storeKey` to an empty string it will rehash. -, it then verifies proof store key hashes to app_hash with simple merkle tree verification method. +For workaround we need to: ++ keep the double hashing and multistore concept for IBC. ++ the `RootStore` will have only two stores: the general one, and another for IBC. `RootStore` should not expose mounting stores in a "runtime" (it's only possible to do it through constructor). ++ The App Hash is a hash of both stores in the `RootStore`. -RootStore merkle proofs, except for IBC module, will have only one pass. For module `M` and it's store key `S_m`, an object `O` with key `k` will be stored in `RootStore.WithPrefix(S_m)`. -This will create a record in `SC` at key `hash(S_m + key)`. - -However this breaks the existing IBC - ### Optimization: compress module keys From 1781fb5944342c042d48ca36efbce580fb877b3f Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 22 Sep 2021 18:15:49 +0200 Subject: [PATCH 12/21] Apply suggestions from code review Co-authored-by: Aleksandr Bezobchuk --- docs/architecture/adr-040-storage-and-smt-state-commitments.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 6e632f5e8abf..8e38aeab91d2 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -149,7 +149,8 @@ In contrast to `MultiStore`, `RootStore` doesn't allow to mount dynamically sub #### Merkle Proofs and IBC -Currently IBC (v1.0) module merkle proof for a `(key, value)` consists of two elements `[storeKey, proof]`. Verification using 2 passes: +Currently the IBC (v1.0) module merkle proofs for a `(key, value)` consists of two elements `[storeKey, proof]`. Verification is done using 2 passes: + 1. Checks that a `proof` is a valid merkle proof `(key, value)` using ICS-23 spec. 2. Then it checks that the `storeKey` and `root(proof)` hashes to the AppHash (App state commitment). From eaa80de93c89876c5bcb03a9d2e4efb385452bd6 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Thu, 23 Sep 2021 11:20:26 +0200 Subject: [PATCH 13/21] reword huffman compression paragraph --- ...dr-040-storage-and-smt-state-commitments.md | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 8e38aeab91d2..6fe4b9606f1d 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -134,7 +134,7 @@ The `WithPrefix` method will create a proxy object for the parent object and wil The following invariant must hold: ``` -for each OP in [Get Has, Set, ...] +for each OP in [Get, Has, Set, ...] store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) ``` @@ -165,19 +165,15 @@ For workaround we need to: ### Optimization: compress module keys -We consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). With Huffman Coding, the `WithPrefix` invariant has the following form: +We consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for the best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). For Merkle Proofs we can't use prefix compression - so it should only apply for the `SS`. Moreover, the prefix compression should be only applied for the module namespace (the first prefix level). More precisely: ++ each module has it's own namespace; ++ when accessing a module namespace we create a KVStore with embedded prefix; ++ that prefix will be compressed only for accessing and managing `SS`. -``` -for each OP in [Get Has, Set, ...] - store.WithPrefix(prefix).OP(key) == store.OP(store.Code(prefix) + key) -``` - -Where `store.Code(prefix)` is a Huffman Code of `prefix` in the given `store`. - -To avoid conflicts in the address space, we need to assure that in the set of prefix codes there are no two elements where one is a prefix of another: +Huffman coding assures that in the set of prefix codes there are no two elements where one is a prefix of another: ``` -for each k1, k2 \in {store.Code(p): p \in StoreModuleKeys} +for each k1, k2 \in {store.HuffmanCode(p): p \in StoreModuleKeys} assert( !k1.hasPrefix(k2) ) ``` From 3f841fd360bf9c3ae902dad364366229f186d2d8 Mon Sep 17 00:00:00 2001 From: Roy Crihfield Date: Fri, 1 Oct 2021 20:26:32 +0800 Subject: [PATCH 14/21] ADR-40: update on multi-store refactor and IBC proofs (#10191) * Update on multistore refactor and IBC proof * cleanup whitespace * Update docs/architecture/adr-040-storage-and-smt-state-commitments.md Co-authored-by: Robert Zaremba * revise for PR * add todo * Update docs/architecture/adr-040-storage-and-smt-state-commitments.md Co-authored-by: Robert Zaremba Co-authored-by: Robert Zaremba --- ...r-040-storage-and-smt-state-commitments.md | 67 ++++++++++++++----- 1 file changed, 51 insertions(+), 16 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 6fe4b9606f1d..ca8ed9f83dae 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -112,13 +112,13 @@ We need to be able to process transactions and roll-back state updates if a tran We identified use-cases, where modules will need to save an object commitment without storing an object itself. Sometimes clients are receiving complex objects, and they have no way to prove a correctness of that object without knowing the storage layout. For those use cases it would be easier to commit to the object without storing it directly. -### Reduce MultiStore +### Refactor MultiStore -Stargate `/store` implementation (StoreV1) adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. +The Stargate `/store` implementation (StoreV1) adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync (see: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). -We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `rootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. +We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `RootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: @@ -138,30 +138,65 @@ for each OP in [Get, Has, Set, ...] store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) ``` +The `RootStore` will have the following interface; the methods for configuring tracing and listeners are omitted for brevity. -The `RootStore` will have the following interface: +```go +// Used where read-only access to versions is needed. +type BasicRootStore interface { + Store + GetKVStore(StoreKey) KVStore + CacheRootStore() CacheRootStore +} +// Used as the main app state, replacing CommitMultiStore. +type CommitRootStore interface { + BasicRootStore + Committer + Snapshotter + + GetVersion(uint64) (BasicRootStore, error) + SetInitialVersion(uint64) error + + ... // Trace and Listen methods +} + +// Replaces CacheMultiStore for branched state. +type CacheRootStore interface { + BasicRootStore + Write() + + ... // Trace and Listen methods +} + +// Example of constructor parameters for the concrete type. +type RootStoreConfig struct { + Upgrades *StoreUpgrades + InitialVersion uint64 + + ReservePrefix(StoreKey, StoreType) +} ``` -TODO -``` + + -In contrast to `MultiStore`, `RootStore` doesn't allow to mount dynamically sub stores. However it can inherit a multistore functionality in it's concrete implementation (which will be needed for IBC support). +In contrast to `MultiStore`, `RootStore` doesn't allow to dynamically mount sub-stores or provide an arbitrary backing DB. -#### Merkle Proofs and IBC +#### Compatibility support + +To ease the transition to this new interface for users, we can create a shim which wraps a `CommitMultiStore` but provides a `CommitRootStore` interface, and expose functions to safely create and access the underlying `CommitMultiStore`. -Currently the IBC (v1.0) module merkle proofs for a `(key, value)` consists of two elements `[storeKey, proof]`. Verification is done using 2 passes: +The new `RootStore` and supporting types can be implemented in a `store/v2` package to avoid breaking existing code. -1. Checks that a `proof` is a valid merkle proof `(key, value)` using ICS-23 spec. -2. Then it checks that the `storeKey` and `root(proof)` hashes to the AppHash (App state commitment). +#### Merkle Proofs and IBC -Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Unfortunately, the straightforward implementation is breaking because all keys in `SC` are hashed. So we can't simply make the last step. Even if we set the `storeKey` to an empty string it will rehash. +Currently, an IBC (v1.0) Merkle proof path consists of two elements (`["", ""]`), with each key corresponding to a separate proof. These are each verified according to individual [ICS-23 specs](https://github.com/cosmos/ibc-go/blob/f7051429e1cf833a6f65d51e6c3df1609290a549/modules/core/23-commitment/types/merkle.go#L17), and the result hash of each step is used as the committed value of the next step, until a root commitment hash is obtained. +The root hash of the proof for `""` is hashed with the `""` to validate against the App Hash. -For workaround we need to: -+ keep the double hashing and multistore concept for IBC. -+ the `RootStore` will have only two stores: the general one, and another for IBC. `RootStore` should not expose mounting stores in a "runtime" (it's only possible to do it through constructor). -+ The App Hash is a hash of both stores in the `RootStore`. +This is not compatible with the `RootStore`, which stores all records in a single Merkle tree structure, and won't produce separate proofs for the store- and record-key. Ideally, the store-key component of the proof could just be omitted, and updated to use a "no-op" spec, so only the record-key is used. However, because the IBC verification code hardcodes the `"ibc"` prefix and applies it to the SDK proof as a separate element of the proof path, this isn't possible without a breaking change. Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module, and updating it across the chains is a time consuming effort. +As a workaround, the `RootStore` will have to maintain two logically separate SMT-based stores: one for IBC state and one for everything else. A simple Merkle map containing these two store keys will act as a `MultiStore`-like index, and the final App hash will be the root hash of this index. +This workaround can be used until the IBC connection code is fully upgraded and supports single-element commitment proofs. ### Optimization: compress module keys From 2a7b1aaf42994b8754620c47f3b4d9f5afd0f4fe Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 6 Oct 2021 19:14:52 +0200 Subject: [PATCH 15/21] review updates --- ...r-040-storage-and-smt-state-commitments.md | 36 ++++++++----------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index ca8ed9f83dae..5f0affd0f27d 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -48,7 +48,7 @@ For data access we propose 2 additional KV buckets (namespaces for the key-value 2. B2: `hash(key, value) → key`: a reverse index to get a key from an SMT path. Recall that SMT will store `(k, v)` as `(hash(k), hash(key, value))`. So, we can get an object value by composing `SMT_path → B2 → B1`. 3. we could use more buckets to optimize the app usage if needed. -We propose to use a KV database for both `SS` and `SC` - each run by its own database instance. This design allows for the separation of `SS` and `SC` into different hardware units, providing support for more complex setup scenarios and improving DB performance (essentially this will create two store shards: one for `SS` and another for `SC`). Moreover, we will be able to configure databases for `SS` and `SC` separately. +We propose to use a KV database for both `SS` and `SC`. The store interface will allow to use the same physical DB backend for both `SS` and `SC` as well two separate DBs. The latter option allows for the separation of `SS` and `SC` into different hardware units, providing support for more complex setup scenarios and improving overall performance: one can use different backends (eg RocksDB and Badger) as well as independently tuning the underlying DB configuration. ### Requirements @@ -120,24 +120,6 @@ The latter indicates, however, that the implementation doesn't provide true modu We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `RootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. -Moreover, to improve usability, we should extend the `KVStore` interface with _prefix store_. This will allow module developers to bind a store to a namespace for module sub-components: - -```go -type KVStore interface { - ... current KVStore - - WithPrefix(prefix string) KVStore -} -``` - -The `WithPrefix` method will create a proxy object for the parent object and will prepend `prefix` to a key parameter of all key-value operations (similar to the `store/prefix.Store` implementation). -The following invariant must hold: - -``` -for each OP in [Get, Has, Set, ...] - store.WithPrefix(prefix).OP(key) == store.OP(prefix + key) -``` - The `RootStore` will have the following interface; the methods for configuring tracing and listeners are omitted for brevity. ```go @@ -181,6 +163,8 @@ type RootStoreConfig struct { In contrast to `MultiStore`, `RootStore` doesn't allow to dynamically mount sub-stores or provide an arbitrary backing DB. +NOTE: modules will be still use a special commitments and their own DBs. For example: a module which will use ZK proofs for state, can store and commit this proofs in the `RootStore` (usually as a single record) and manage the specialized store privately or using the `SS` low level interface. + #### Compatibility support To ease the transition to this new interface for users, we can create a shim which wraps a `CommitMultiStore` but provides a `CommitRootStore` interface, and expose functions to safely create and access the underlying `CommitMultiStore`. @@ -192,11 +176,11 @@ The new `RootStore` and supporting types can be implemented in a `store/v2` pack Currently, an IBC (v1.0) Merkle proof path consists of two elements (`["", ""]`), with each key corresponding to a separate proof. These are each verified according to individual [ICS-23 specs](https://github.com/cosmos/ibc-go/blob/f7051429e1cf833a6f65d51e6c3df1609290a549/modules/core/23-commitment/types/merkle.go#L17), and the result hash of each step is used as the committed value of the next step, until a root commitment hash is obtained. The root hash of the proof for `""` is hashed with the `""` to validate against the App Hash. -This is not compatible with the `RootStore`, which stores all records in a single Merkle tree structure, and won't produce separate proofs for the store- and record-key. Ideally, the store-key component of the proof could just be omitted, and updated to use a "no-op" spec, so only the record-key is used. However, because the IBC verification code hardcodes the `"ibc"` prefix and applies it to the SDK proof as a separate element of the proof path, this isn't possible without a breaking change. Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module, and updating it across the chains is a time consuming effort. +This is not compatible with the `RootStore`, which stores all records in a single Merkle tree structure, and won't produce separate proofs for the store- and record-key. Ideally, the store-key component of the proof could just be omitted, and updated to use a "no-op" spec, so only the record-key is used. However, because the IBC verification code hardcodes the `"ibc"` prefix and applies it to the SDK proof as a separate element of the proof path, this isn't possible without a breaking change. Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Requesting an update of the IBC module across the chains is a time consuming effort and not easily feasible. -As a workaround, the `RootStore` will have to maintain two logically separate SMT-based stores: one for IBC state and one for everything else. A simple Merkle map containing these two store keys will act as a `MultiStore`-like index, and the final App hash will be the root hash of this index. +As a workaround, the `RootStore` will have to use two separate SMTs (they could use the same underlying DB): one for IBC state and one for everything else. A simple Merkle map that reference these SMTs will act as a Merkle Tree to create a final App hash. The Merkle map is not stored in a DBs - it's constructed in the runtime. -This workaround can be used until the IBC connection code is fully upgraded and supports single-element commitment proofs. +The solution presented here can be used until the IBC module is fully upgraded to supports single-element commitment proofs. ### Optimization: compress module keys @@ -214,6 +198,14 @@ for each k1, k2 \in {store.HuffmanCode(p): p \in StoreModuleKeys} NOTE: We need to assure that the codes won't change. Huffman Coding depends on the keys and its frequency - so we would need to generate the codes and then fix the mapping in a static variable. +#### Alternatives + ++ Single byte prefix ++ Double byte prefix ++ Varint prefix (varint is a simplified version of Huffman Coding without taking the frequency information into account). + +TODO: need to make decision about the key compression. + ## Consequences ### Backwards Compatibility From fbe46760afa75a78aa374da95177cb60e4351b81 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 6 Oct 2021 19:21:07 +0200 Subject: [PATCH 16/21] add todo for protobuf message type compression --- .../adr-040-storage-and-smt-state-commitments.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index c8a98b6c2fb4..9027fa2a49ef 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -214,6 +214,12 @@ NOTE: We need to assure that the codes won't change. Huffman Coding depends on t TODO: need to make decision about the key compression. +### Optimization: SS key compression + +Some objects may be saved with key, which contains a Protobuf message type. Such keys are long. We could save a lot of space if we can map Protobuf message types in varints. + +TODO: finalize this or move to another ADR. + ## Consequences ### Backwards Compatibility From 5dad4ef698cc7fca90ef66b3d97c25451587010a Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 6 Oct 2021 19:32:57 +0200 Subject: [PATCH 17/21] add link to a discussion --- .../adr-040-storage-and-smt-state-commitments.md | 12 ++++-------- 1 file changed, 4 insertions(+), 8 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 9027fa2a49ef..7e0161011dce 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -122,13 +122,9 @@ We identified use-cases, where modules will need to save an object commitment wi ### Refactor MultiStore -The Stargate `/store` implementation (StoreV1) adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. +The Stargate `/store` implementation (store/v1) adds an additional layer in the SDK store construction - the `MultiStore` structure. The multistore exists to support the modularity of the Cosmos SDK - each module is using its own instance of IAVL, but in the current implementation, all instances share the same database. The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and atomic DB commits (see: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370) and [discussion](https://github.com/cosmos/cosmos-sdk/discussions/8297#discussioncomment-757043)). -The latter indicates, however, that the implementation doesn't provide true modularity. Instead it causes problems related to race condition and sync (see: [\#6370](https://github.com/cosmos/cosmos-sdk/issues/6370)). - -We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `RootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. - -The `RootStore` will have the following interface; the methods for configuring tracing and listeners are omitted for brevity. +We propose to reduce the multistore concept from the SDK, and to use a single instance of `SC` and `SS` in a `RootStore` object. To avoid confusion, we should rename the `MultiStore` interface to `RootStore`. The `RootStore` will have the following interface; the methods for configuring tracing and listeners are omitted for brevity. ```go // Used where read-only access to versions is needed. @@ -190,7 +186,7 @@ As a workaround, the `RootStore` will have to use two separate SMTs (they could The solution presented here can be used until the IBC module is fully upgraded to supports single-element commitment proofs. -### Optimization: compress module keys +### Optimization: compress module key prefixes We consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for the best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). For Merkle Proofs we can't use prefix compression - so it should only apply for the `SS`. Moreover, the prefix compression should be only applied for the module namespace (the first prefix level). More precisely: + each module has it's own namespace; @@ -214,7 +210,7 @@ NOTE: We need to assure that the codes won't change. Huffman Coding depends on t TODO: need to make decision about the key compression. -### Optimization: SS key compression +## Optimization: SS key compression Some objects may be saved with key, which contains a Protobuf message type. Such keys are long. We could save a lot of space if we can map Protobuf message types in varints. From f9c77ae92798da46515d87c9abffe4a997878904 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 6 Oct 2021 19:36:52 +0200 Subject: [PATCH 18/21] guarantee atomic commit with IBC workaround proposal --- .../architecture/adr-040-storage-and-smt-state-commitments.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 7e0161011dce..c2b3aecf2991 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -184,7 +184,9 @@ This is not compatible with the `RootStore`, which stores all records in a singl As a workaround, the `RootStore` will have to use two separate SMTs (they could use the same underlying DB): one for IBC state and one for everything else. A simple Merkle map that reference these SMTs will act as a Merkle Tree to create a final App hash. The Merkle map is not stored in a DBs - it's constructed in the runtime. -The solution presented here can be used until the IBC module is fully upgraded to supports single-element commitment proofs. +The workaround can still guarantee atomic syncs: the [proposed DB backends](#evaluated-kv-databases) support atomic transactions and efficient rollbacks, which will be used in the commit phase. + +The presented workaround can be used until the IBC module is fully upgraded to supports single-element commitment proofs. ### Optimization: compress module key prefixes From 1fec94ca25519b019777afbfb0960345c5c57899 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Wed, 6 Oct 2021 21:00:55 +0200 Subject: [PATCH 19/21] adding more links to references --- .../architecture/adr-040-storage-and-smt-state-commitments.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index c2b3aecf2991..4d07ed9d66ca 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -182,7 +182,7 @@ The root hash of the proof for `""` is hashed with the `" This is not compatible with the `RootStore`, which stores all records in a single Merkle tree structure, and won't produce separate proofs for the store- and record-key. Ideally, the store-key component of the proof could just be omitted, and updated to use a "no-op" spec, so only the record-key is used. However, because the IBC verification code hardcodes the `"ibc"` prefix and applies it to the SDK proof as a separate element of the proof path, this isn't possible without a breaking change. Breaking this behavior would severely impact the Cosmos ecosystem which already widely adopts the IBC module. Requesting an update of the IBC module across the chains is a time consuming effort and not easily feasible. -As a workaround, the `RootStore` will have to use two separate SMTs (they could use the same underlying DB): one for IBC state and one for everything else. A simple Merkle map that reference these SMTs will act as a Merkle Tree to create a final App hash. The Merkle map is not stored in a DBs - it's constructed in the runtime. +As a workaround, the `RootStore` will have to use two separate SMTs (they could use the same underlying DB): one for IBC state and one for everything else. A simple Merkle map that reference these SMTs will act as a Merkle Tree to create a final App hash. The Merkle map is not stored in a DBs - it's constructed in the runtime. The IBC substore key must be `"ibc"`. The workaround can still guarantee atomic syncs: the [proposed DB backends](#evaluated-kv-databases) support atomic transactions and efficient rollbacks, which will be used in the commit phase. @@ -273,3 +273,5 @@ We were discussing use case where modules can use a support database, which is n + Facebook Diem (Libra) SMT [design](https://developers.diem.com/papers/jellyfish-merkle-tree/2021-01-14.pdf) + [Trillian Revocation Transparency](https://github.com/google/trillian/blob/master/docs/papers/RevocationTransparency.pdf), [Trillian Verifiable Data Structures](https://github.com/google/trillian/blob/master/docs/papers/VerifiableDataStructures.pdf). + Design and implementation [discussion](https://github.com/cosmos/cosmos-sdk/discussions/8297). ++ [How to Upgrade IBC Chains and their Clients](https://github.com/cosmos/ibc-go/blob/main/docs/ibc/upgrades/quick-guide.md) ++ [ADR-40 Effect on IBC](https://github.com/cosmos/ibc-go/discussions/256) From 74193a96f81edb642cc87869dc534fa0e96eb330 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Thu, 21 Oct 2021 14:55:29 +0200 Subject: [PATCH 20/21] Apply suggestions from code review Co-authored-by: Roy Crihfield --- .../architecture/adr-040-storage-and-smt-state-commitments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index 4d07ed9d66ca..ccb667861317 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -165,9 +165,9 @@ type RootStoreConfig struct { -In contrast to `MultiStore`, `RootStore` doesn't allow to dynamically mount sub-stores or provide an arbitrary backing DB. +In contrast to `MultiStore`, `RootStore` doesn't allow to dynamically mount sub-stores or provide an arbitrary backing DB for individual sub-stores. -NOTE: modules will be still use a special commitments and their own DBs. For example: a module which will use ZK proofs for state, can store and commit this proofs in the `RootStore` (usually as a single record) and manage the specialized store privately or using the `SS` low level interface. +NOTE: modules will be able to use a special commitment and their own DBs. For example: a module which will use ZK proofs for state can store and commit this proof in the `RootStore` (usually as a single record) and manage the specialized store privately or using the `SC` low level interface. #### Compatibility support From 2390332ad43e061a631d94f3f0e04024148f1002 Mon Sep 17 00:00:00 2001 From: Robert Zaremba Date: Thu, 21 Oct 2021 15:02:00 +0200 Subject: [PATCH 21/21] reword the module key compression part --- ...r-040-storage-and-smt-state-commitments.md | 20 ++++--------------- 1 file changed, 4 insertions(+), 16 deletions(-) diff --git a/docs/architecture/adr-040-storage-and-smt-state-commitments.md b/docs/architecture/adr-040-storage-and-smt-state-commitments.md index ccb667861317..c97cc15ed998 100644 --- a/docs/architecture/adr-040-storage-and-smt-state-commitments.md +++ b/docs/architecture/adr-040-storage-and-smt-state-commitments.md @@ -162,6 +162,7 @@ type RootStoreConfig struct { ReservePrefix(StoreKey, StoreType) } ``` + @@ -190,25 +191,12 @@ The presented workaround can be used until the IBC module is fully upgraded to s ### Optimization: compress module key prefixes -We consider a compression of prefix keys using [Huffman Coding](https://en.wikipedia.org/wiki/Huffman_coding). It will require knowledge of used prefixes (module store keys) a priori. And for the best results, it will need frequency information for each prefix (how often objects are stored in the store under the same prefix key). For Merkle Proofs we can't use prefix compression - so it should only apply for the `SS`. Moreover, the prefix compression should be only applied for the module namespace (the first prefix level). More precisely: +We consider a compression of prefix keys by creating a mapping from module key to an integer, and serializing the integer using varint coding. Varint coding assures that different values don't have common byte prefix. For Merkle Proofs we can't use prefix compression - so it should only apply for the `SS` keys. Moreover, the prefix compression should be only applied for the module namespace. More precisely: + each module has it's own namespace; + when accessing a module namespace we create a KVStore with embedded prefix; -+ that prefix will be compressed only for accessing and managing `SS`. - -Huffman coding assures that in the set of prefix codes there are no two elements where one is a prefix of another: - -``` -for each k1, k2 \in {store.HuffmanCode(p): p \in StoreModuleKeys} - assert( !k1.hasPrefix(k2) ) -``` - -NOTE: We need to assure that the codes won't change. Huffman Coding depends on the keys and its frequency - so we would need to generate the codes and then fix the mapping in a static variable. - -#### Alternatives ++ that prefix will be compressed only when accessing and managing `SS`. -+ Single byte prefix -+ Double byte prefix -+ Varint prefix (varint is a simplified version of Huffman Coding without taking the frequency information into account). +We need to assure that the codes won't change. We can fix the mapping in a static variable (provided by an app) or SS state under a special key. TODO: need to make decision about the key compression.