Introduce trie level cache and remove state cache (#11407)

* trie state cache

* Also cache missing access on read.

* fix comp

* bis

* fix

* use has_lru

* remove local storage cache on size 0.

* No cache.

* local cache only

* trie cache and local cache

* storage cache (with local)

* trie cache no local cache

* Add state access benchmark

* Remove warnings etc

* Add trie cache benchmark

* No extra "clone" required

* Change benchmark to use multiple blocks

* Use patches

* Integrate shitty implementation

* More stuff

* Revert "Merge branch 'master' into trie_state_cache"

This reverts commit 2c1886d, reversing
changes made to 540b4fd.

* Improve benchmark

* Adapt to latest changes

* Adapt to changes in trie

* Add a test that uses iterator

* Start fixing it

* Remove obsolete file

* Make it compile

* Start rewriting the trie node cache

* More work on the cache

* More docs and code etc

* Make data cache an optional

* Tests

* Remove debug stuff

* Recorder

* Some docs and a simple test for the recorder

* Compile fixes

* Make it compile

* More fixes

* More fixes

* Fix fix fix

* Make sure cache and recorder work together for basic stuff

* Test that data caching and recording works

* Test `TrieDBMut` with caching

* Try something

* Fixes, fixes, fixes

* Forward the recorder

* Make it compile

* Use recorder in more places

* Switch to new `with_optional_recorder` fn

* Refactor and cleanups

* Move `ProvingBackend` tests

* Simplify

* Move over all functionality to the essence

* Fix compilation

* Implement estimate encoded size for StorageProof

* Start using the `cache` everywhere

* Use the cache everywhere

* Fix compilation

* Fix tests

* Adds `TrieBackendBuilder` and enhances the tests

* Ensure that recorder drain checks that values are found as expected

* Switch over to `TrieBackendBuilder`

* Start fixing the problem with child tries and recording

* Fix recording of child tries

* Make it compile

* Overwrite `storage_hash` in `TrieBackend`

* Add `storage_cache` to  the benchmarks

* Fix `no_std` build

* Speed up cache lookup

* Extend the state access benchmark to also hash a runtime

* Fix build

* Fix compilation

* Rewrite value cache

* Add lru cache

* Ensure that the cache lru works

* Value cache should not be optional

* Add support for keeping the shared node cache in its bounds

* Make the cache configurable

* Check that the cache respects the bounds

* Adds a new test

* Fixes

* Docs and some renamings

* More docs

* Start using the new recorder

* Fix more code

* Take `self` argument

* Remove warnings

* Fix benchmark

* Fix accounting

* Rip off the state cache

* Start fixing fallout after removing the state cache

* Make it compile after trie changes

* Fix test

* Add some logging

* Some docs

* Some fixups and clean ups

* Fix benchmark

* Remove unneeded file

* Use git for patching

* Make CI happy

* Update primitives/trie/Cargo.toml

Co-authored-by: Koute <koute@users.noreply.github.com>

* Update primitives/state-machine/src/trie_backend.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Introduce new `AsTrieBackend` trait

* Make the LocalTrieCache not clonable

* Make it work in no_std and add docs

* Remove duplicate dependency

* Switch to ahash for better performance

* Speedup value cache merge

* Output errors on underflow

* Ensure the internal LRU map doesn't grow too much

* Use const fn to calculate the value cache element size

* Remove cache configuration

* Fix

* Clear the cache in between for more testing

* Try to come up with a failing test case

* Make the test fail

* Fix the child trie recording

* Make everything compile after the changes to trie

* Adapt to latest trie-db changes

* Fix on stable

* Update primitives/trie/src/cache.rs

Co-authored-by: cheme <emericchevalier.pro@gmail.com>

* Fix wrong merge

* Docs

* Fix warnings

* Cargo.lock

* Bump pin-project

* Fix warnings

* Switch to released crate version

* More fixes

* Make clippy and rustdocs happy

* More clippy

* Print error when using deprecated `--state-cache-size`

* 🤦

* Fixes

* Fix storage_hash linkings

* Update client/rpc/src/dev/mod.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Review feedback

* encode bound

* Rework the shared value cache

Instead of using a `u64` to represent the key we now use an `Arc<[u8]>`. This arc is also stored in
some extra `HashSet`. We store the key are in an extra `HashSet` to de-duplicate the keys accross
different storage roots. When the latest key usage is dropped in the lru, we also remove the key
from the `HashSet`.

* Improve of the cache by merging the old and new solution

* FMT

* Please stop coming back all the time :crying:

* Update primitives/trie/src/cache/shared_cache.rs

Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

* Fixes

* Make clippy happy

* Ensure we don't deadlock

* Only use one lock to simplify the code

* Do not depend on `Hasher`

* Fix tests

* FMT

* Clippy 🤦

Co-authored-by: cheme <emericchevalier.pro@gmail.com>
Co-authored-by: Koute <koute@users.noreply.github.com>
Co-authored-by: Arkadiy Paronyan <arkady.paronyan@gmail.com>

Author: 4 people

Cargo.lock

@@ -20,7 +20,7 @@ use std::{collections::HashMap, sync::Arc};
 
 use kvdb::KeyValueDB;
 use node_primitives::Hash;
-use sp_trie::{trie_types::TrieDBMutV1, TrieMut};
+use sp_trie::{trie_types::TrieDBMutBuilderV1, TrieMut};
 
 use crate::simple_trie::SimpleTrie;
 
@@ -43,7 +43,8 @@ pub fn generate_trie(
 		);
 		let mut trie = SimpleTrie { db, overlay: &mut overlay };
 		{
-			let mut trie_db = TrieDBMutV1::<crate::simple_trie::Hasher>::new(&mut trie, &mut root);
+			let mut trie_db =
+				TrieDBMutBuilderV1::<crate::simple_trie::Hasher>::new(&mut trie, &mut root).build();
 			for (key, value) in key_values {
 				trie_db.insert(&key, &value).expect("trie insertion failed");
 			}

bin/node/bench/src/generator.rs

@@ -23,7 +23,7 @@ use kvdb::KeyValueDB;
 use lazy_static::lazy_static;
 use rand::Rng;
 use sp_state_machine::Backend as _;
-use sp_trie::{trie_types::TrieDBMutV1, TrieMut as _};
+use sp_trie::{trie_types::TrieDBMutBuilderV1, TrieMut as _};
 use std::{borrow::Cow, collections::HashMap, sync::Arc};
 
 use node_primitives::Hash;
@@ -180,7 +180,7 @@ impl core::Benchmark for TrieReadBenchmark {
 		let storage: Arc<dyn sp_state_machine::Storage<sp_core::Blake2Hasher>> =
 			Arc::new(Storage(db.open(self.database_type)));
 
-		let trie_backend = sp_state_machine::TrieBackend::new(storage, self.root);
+		let trie_backend = sp_state_machine::TrieBackendBuilder::new(storage, self.root).build();
 		for (warmup_key, warmup_value) in self.warmup_keys.iter() {
 			let value = trie_backend
 				.storage(&warmup_key[..])
@@ -286,8 +286,7 @@ impl core::Benchmark for TrieWriteBenchmark {
 
 		let mut overlay = HashMap::new();
 		let mut trie = SimpleTrie { db: kvdb.clone(), overlay: &mut overlay };
-		let mut trie_db_mut = TrieDBMutV1::from_existing(&mut trie, &mut new_root)
-			.expect("Failed to create TrieDBMut");
+		let mut trie_db_mut = TrieDBMutBuilderV1::from_existing(&mut trie, &mut new_root).build();
 
 		for (warmup_key, warmup_value) in self.warmup_keys.iter() {
 			let value = trie_db_mut

bin/node/bench/src/trie.rs

@@ -72,8 +72,7 @@ fn new_node(tokio_handle: Handle) -> node_cli::service::NewFullBase {
 		keystore: KeystoreConfig::InMemory,
 		keystore_remote: Default::default(),
 		database: DatabaseSource::RocksDb { path: root.join("db"), cache_size: 128 },
-		state_cache_size: 67108864,
-		state_cache_child_ratio: None,
+		trie_cache_maximum_size: Some(64 * 1024 * 1024),
 		state_pruning: Some(PruningMode::ArchiveAll),
 		blocks_pruning: BlocksPruning::All,
 		chain_spec: spec,

bin/node/cli/benches/block_production.rs

@@ -66,8 +66,7 @@ fn new_node(tokio_handle: Handle) -> node_cli::service::NewFullBase {
 		keystore: KeystoreConfig::InMemory,
 		keystore_remote: Default::default(),
 		database: DatabaseSource::RocksDb { path: root.join("db"), cache_size: 128 },
-		state_cache_size: 67108864,
-		state_cache_child_ratio: None,
+		trie_cache_maximum_size: Some(64 * 1024 * 1024),
 		state_pruning: Some(PruningMode::ArchiveAll),
 		blocks_pruning: BlocksPruning::All,
 		chain_spec: spec,

bin/node/cli/benches/transaction_pool.rs

@@ -388,8 +388,7 @@ impl BenchDb {
 		keyring: &BenchKeyring,
 	) -> (Client, std::sync::Arc<Backend>, TaskExecutor) {
 		let db_config = sc_client_db::DatabaseSettings {
-			state_cache_size: 16 * 1024 * 1024,
-			state_cache_child_ratio: Some((0, 100)),
+			trie_cache_maximum_size: Some(16 * 1024 * 1024),
 			state_pruning: Some(PruningMode::ArchiveAll),
 			source: database_type.into_settings(dir.into()),
 			blocks_pruning: sc_client_db::BlocksPruning::All,

bin/node/testing/src/bench.rs

@@ -32,7 +32,8 @@ use sp_runtime::{
 	Justification, Justifications, StateVersion, Storage,
 };
 use sp_state_machine::{
-	ChildStorageCollection, IndexOperation, OffchainChangesCollection, StorageCollection,
+	backend::AsTrieBackend, ChildStorageCollection, IndexOperation, OffchainChangesCollection,
+	StorageCollection,
 };
 use sp_storage::{ChildInfo, StorageData, StorageKey};
 use std::collections::{HashMap, HashSet};
@@ -448,7 +449,12 @@ pub trait Backend<Block: BlockT>: AuxStore + Send + Sync {
 	/// Associated blockchain backend type.
 	type Blockchain: BlockchainBackend<Block>;
 	/// Associated state backend type.
-	type State: StateBackend<HashFor<Block>> + Send;
+	type State: StateBackend<HashFor<Block>>
+		+ Send
+		+ AsTrieBackend<
+			HashFor<Block>,
+			TrieBackendStorage = <Self::State as StateBackend<HashFor<Block>>>::TrieBackendStorage,
+		>;
 	/// Offchain workers local storage.
 	type OffchainStorage: OffchainStorage;
 

client/api/src/backend.rs

@@ -855,10 +855,18 @@ mod tests {
 			.expect("header get error")
 			.expect("there should be header");
 
-		let extrinsics_num = 4;
-		let extrinsics = (0..extrinsics_num)
-			.map(|v| Extrinsic::IncludeData(vec![v as u8; 10]))
-			.collect::<Vec<_>>();
+		let extrinsics_num = 5;
+		let extrinsics = std::iter::once(
+			Transfer {
+				from: AccountKeyring::Alice.into(),
+				to: AccountKeyring::Bob.into(),
+				amount: 100,
+				nonce: 0,
+			}
+			.into_signed_tx(),
+		)
+		.chain((0..extrinsics_num - 1).map(|v| Extrinsic::IncludeData(vec![v as u8; 10])))
+		.collect::<Vec<_>>();
 
 		let block_limit = genesis_header.encoded_size() +
 			extrinsics
@@ -922,8 +930,9 @@ mod tests {
 		.unwrap();
 
 		// The block limit didn't changed, but we now include the proof in the estimation of the
-		// block size and thus, one less transaction should fit into the limit.
-		assert_eq!(block.extrinsics().len(), extrinsics_num - 2);
+		// block size and thus, only the `Transfer` will fit into the block. It reads more data
+		// than we have reserved in the block limit.
+		assert_eq!(block.extrinsics().len(), 1);
 	}
 
 	#[test]

client/basic-authorship/src/basic_authorship.rs

@@ -73,8 +73,7 @@ impl ChainInfoCmd {
 		B: BlockT,
 	{
 		let db_config = sc_client_db::DatabaseSettings {
-			state_cache_size: config.state_cache_size,
-			state_cache_child_ratio: config.state_cache_child_ratio.map(|v| (v, 100)),
+			trie_cache_maximum_size: config.trie_cache_maximum_size,
 			state_pruning: config.state_pruning.clone(),
 			source: config.database.clone(),
 			blocks_pruning: config.blocks_pruning,

client/cli/src/commands/chain_info_cmd.rs

@@ -230,18 +230,12 @@ pub trait CliConfiguration<DCV: DefaultConfigurationValues = ()>: Sized {
 		})
 	}
 
-	/// Get the state cache size.
+	/// Get the trie cache maximum size.
 	///
 	/// By default this is retrieved from `ImportParams` if it is available. Otherwise its `0`.
-	fn state_cache_size(&self) -> Result<usize> {
-		Ok(self.import_params().map(|x| x.state_cache_size()).unwrap_or_default())
-	}
-
-	/// Get the state cache child ratio (if any).
-	///
-	/// By default this is `None`.
-	fn state_cache_child_ratio(&self) -> Result<Option<usize>> {
-		Ok(Default::default())
+	/// If `None` is returned the trie cache is disabled.
+	fn trie_cache_maximum_size(&self) -> Result<Option<usize>> {
+		Ok(self.import_params().map(|x| x.trie_cache_maximum_size()).unwrap_or_default())
 	}
 
 	/// Get the state pruning mode.
@@ -533,8 +527,7 @@ pub trait CliConfiguration<DCV: DefaultConfigurationValues = ()>: Sized {
 			keystore_remote,
 			keystore,
 			database: self.database_config(&config_dir, database_cache_size, database)?,
-			state_cache_size: self.state_cache_size()?,
-			state_cache_child_ratio: self.state_cache_child_ratio()?,
+			trie_cache_maximum_size: self.trie_cache_maximum_size()?,
 			state_pruning: self.state_pruning()?,
 			blocks_pruning: self.blocks_pruning()?,
 			wasm_method: self.wasm_method()?,

client/cli/src/config.rs

@@ -95,14 +95,30 @@ pub struct ImportParams {
 	pub execution_strategies: ExecutionStrategiesParams,
 
 	/// Specify the state cache size.
+	///
+	/// Providing `0` will disable the cache.
 	#[clap(long, value_name = "Bytes", default_value = "67108864")]
-	pub state_cache_size: usize,
+	pub trie_cache_size: usize,
+
+	/// DEPRECATED
+	///
+	/// Switch to `--trie-cache-size`.
+	#[clap(long)]
+	state_cache_size: Option<usize>,
 }
 
 impl ImportParams {
-	/// Specify the state cache size.
-	pub fn state_cache_size(&self) -> usize {
-		self.state_cache_size
+	/// Specify the trie cache maximum size.
+	pub fn trie_cache_maximum_size(&self) -> Option<usize> {
+		if self.state_cache_size.is_some() {
+			eprintln!("`--state-cache-size` was deprecated. Please switch to `--trie-cache-size`.");
+		}
+
+		if self.trie_cache_size == 0 {
+			None
+		} else {
+			Some(self.trie_cache_size)
+		}
 	}
 
 	/// Get the WASM execution method from the parameters

client/cli/src/params/import_params.rs

@@ -35,9 +35,12 @@ sp-state-machine = { version = "0.12.0", path = "../../primitives/state-machine"
 sp-trie = { version = "6.0.0", path = "../../primitives/trie" }
 
 [dev-dependencies]
+criterion = "0.3.3"
 kvdb-rocksdb = "0.15.1"
+rand = "0.8.4"
+tempfile = "3.1.0"
 quickcheck = { version = "1.0.3", default-features = false }
-tempfile = "3"
+kitchensink-runtime = { path = "../../bin/node/runtime" }
 sp-tracing = { version = "5.0.0", path = "../../primitives/tracing" }
 substrate-test-runtime-client = { version = "2.0.0", path = "../../test-utils/runtime/client" }
 
@@ -46,3 +49,10 @@ default = []
 test-helpers = []
 runtime-benchmarks = []
 rocksdb = ["kvdb-rocksdb"]
+
+[[bench]]
+name = "state_access"
+harness = false
+
+[lib]
+bench = false