Merge pull request #226 from idky137/update_docs_2

update docs

README.md

@@ -29,10 +29,11 @@ Currently Zebra's `ReadStateService` only enables direct access to chain data (b
 
 ## Documentation
 - [Use Cases](./docs/use_cases.md): Holds instructions and example use cases.
-- [Testing](./docs/testing.md): Hold instructions fo running tests.
-- [Live Service System Architecture](./docs/live_system_architecture.pdf): Holds the Zcash system architecture diagram for the Zaino live service.
-- [Library System Architecture](./docs/lib_system_architecture.pdf): Holds the Zcash system architecture diagram for the Zaino client library.
-- [Internal Architecture](./docs/internal_architecture.pdf): Holds an internal Zaino system architecture diagram.
+- [Testing](./docs/testing.md): Holds instructions for running tests.
+- [Live Service System Architecture](./docs/zaino_live_system_architecture.pdf): Holds the Zcash system architecture diagram for the Zaino live service.
+- [Library System Architecture](./docs/zaino_lib_system_architecture.pdf): Holds the Zcash system architecture diagram for the Zaino client library.
+- [ZainoD (Live Service) Internal Architecture](./docs/zaino_serve_architecture_v020.pdf): Holds an internal Zaino system architecture diagram.
+- [Zaino-State (Library) Internal Architecture](./docs/zaino_state_architecture_v020.pdf): Holds an internal Zaino system architecture diagram.
 - [Internal Specification](./docs/internal_spec.md): Holds a specification for Zaino and its crates, detailing their functionality, interfaces and dependencies.
 - [RPC API Spec](./docs/rpc_api.md): Holds a full specification of all of the RPC services served by Zaino.
 - [Cargo Docs](https://zingolabs.github.io/zaino/): Holds a full code specification for Zaino.

docs/internal_spec.md

@@ -1,21 +1,91 @@
 # Zaino
 The Zaino repo consists of several crates that collectively provide an indexing service and APIs for the Zcash blockchain. The crates are modularized to separate concerns, enhance maintainability, and allow for flexible integration.
 
-The main crates are:
+### Crates
   - `Zainod`
   - `Zaino-Serve`
   - `Zaino-State`
   - `Zaino-Fetch`
   - `Zaino-Proto`
+  - `Zaino-Testutils`
+  - `Integration-tests`
 
 ### Workspace Dependencies
-  - `zcash_protocol`
-  - `zebra-chain`
-  - `zebra-rpc`
-  - `tokio`
-  - `tonic`
-  - `http`
-  - `thiserror`
+**Zingo Labs:**
+- zingolib
+- testvectors
+
+**Librustzcash:**
+- zcash_client_backend
+- zcash_protocol
+
+
+**Zebra:**
+- zebra-chain
+- zebra-state
+- zebra-rpc
+
+
+**Zingo-infra-testutils:**
+- zingo-infra-testutils
+- zingo-infra-services
+
+**Runtime:**
+- tokio
+- tokio-stream 
+
+**CLI:**
+- clap
+
+**Tracing:**
+- tracing 
+- tracing-subscriber
+- tracing-futures
+
+**Network / RPC:**
+- http
+- url
+- reqwest
+- tower
+- tonic
+- tonic-build
+- prost
+- serde 
+- serde_json
+- jsonrpc-core 
+- jsonrpsee-types
+
+**Hashmaps, channels, DBs:**
+- indexmap 
+- crossbeam-channel
+- dashmap 
+- lmdb 
+
+**Async:**
+- async-stream 
+- async-trait 
+- futures 
+
+**Utility:**
+- thiserror
+- lazy-regex
+- once_cell
+- ctrlc 
+- chrono
+- which 
+- whoami 
+
+**Formats:**
+- base64 
+- byteorder
+- sha2
+- hex
+- toml
+
+**Test:**
+- portpicker
+- tempfile
+
 
 Below is a detailed specification for each crate.
 
@@ -35,24 +105,6 @@ A full specification of the public functionality and RPC services available in Z
   - Manages the asynchronous runtime using `Tokio`.
   - Handles graceful shutdowns and restarts.
 
-### Interfaces
-- Executable Interface:
-  - Provides a CLI for configuring the service. (Currently it is only possible to set the conf file path.)
-
-- Configuration Files:
-  - Supports TOML files for complex configurations.
-
-### Dependencies
-  - `zaino-fetch`
-  - `zaino-serve`
-  - `tokio`
-  - `http`
-  - `thiserror`
-  - `serde`
-  - `ctrlc`
-  - `toml`
-  - `clap`
-
 Full documentation for `ZainoD` can be found [here](https://zingolabs.github.io/zaino/zainod/index.html) and [here](https://zingolabs.github.io/zaino/zainodlib/index.html).
 
 
@@ -62,51 +114,24 @@ Full documentation for `ZainoD` can be found [here](https://zingolabs.github.io/
 ### Functionality
 - gRPC Server Implementation:
   - Utilizes `Tonic` to implement the gRPC server.
-  - Uses a `Director-Ingestor-Worker` model (see [Internal Architecture](./internal_architecture.pdf)) to allow the addition of Nym or Tor based `Ingestors`.
-  - Dynamically manages the internal Worker pool and Request queue and active Ingestors, handling errors and restarting services where necessary.
   - Hosts the `CompactTxStreamerServer` service for client interactions.
 
 - `CompactTxStreamerServer` Method Implementations:
   - Implements the full set of methods as defined in the [LightWallet Protocol](https://github.com/zcash/librustzcash/blob/main/zcash_client_backend/proto/service.proto).
 
 - Request Handling:
   - Validates and parses client requests.
-  - Communicates with `zaino-state` or `zaino-fetch` to retrieve data.
+  - Communicates with `zaino-state` to retrieve data.
 
 - Error Handling:
   - Maps internal errors to appropriate gRPC status codes.
   - Provides meaningful error messages to clients.
 
-### Interfaces
-- Public gRPC API:
-  - Defined in `zaino-proto` and exposed to clients.
-
-- Internal Library:
-  - The `server::director` module provides the following gRPC server management functions: `ServerStatus::new`, `ServerStatus::load`, `Server::spawn`, `Server::serve`, `Server::check_for_shutdown`, `Server::shutdown`, `Server::status`, `Server::statustype`, `Server::statuses`, `Server::check_statuses`.
-
-### Dependencies
-  - `zaino-proto`
-  - `zaino-fetch`
-  - `zebra-chain`
-  - `zebra-rpc`
-  - `tokio`
-  - `tonic`
-  - `http`
-  - `thiserror`
-  - `prost`
-  - `hex`
-  - `tokio-stream`
-  - `futures`
-  - `async-stream`
-  - `crossbeam-channel`
-  - `lazy-regex`
-  - `whoami`
-
 Full documentation for `Zaino-Serve` can be found [here](https://zingolabs.github.io/zaino/zaino_serve/index.html).
 
 
 ## Zaino-State
-`Zaino-State` is a library that provides access to the mempool and blockchain data by interfacing directly with `Zebra`'s `ReadStateService`. It is designed for direct consumption by full node wallets and internal services. (Currently unimplemented.)
+`Zaino-State` is Zaino's chain fetch and transaction submission library, interfacing with zcash validators throught a configurable backend. It is designed for direct consumption by full node wallets and internal services, enabling a simlified interface for Zcash clients.
 
 ### Functionality
 - Blockchain Data Access:
@@ -122,27 +147,17 @@ Full documentation for `Zaino-Serve` can be found [here](https://zingolabs.githu
   - Keeps track of the chain state in sync with Zebra.
   - Handles reorgs and updates to the best chain.
 
-Caching Mechanisms:
+- Caching Mechanisms:
   - Implements caching for frequently accessed data to improve performance.
 
-### Interfaces
-- Public Library API:
-  - Provides data retrieval and submission functions that directly correspond to the RPC services offered by `zaino-serve`.
-  - Provides asynchronous interfaces compatible with `Tokio`.
-
-- Event Streams:
-  - Offers highly concurrent, lock-free streams or channels to subscribe to blockchain events.
-
-### Dependencies
-  - `zebra-state`
-  - `tokio`
-  - `thiserror`
+- Configurable Backend:
+  - Implementes a configurable backend service enabling clients to use a single interface for any validator set-up.
 
 Full documentation for `Zaino-State` can be found [here](https://zingolabs.github.io/zaino/zaino_state/index.html).
 
 
 ## Zaino-Fetch
-`Zaino-Fetch` is a library that provides access to the mempool and blockchain data using Zebra's RPC interface. It is primarily used as a backup and for backward compatibility with systems that rely on RPC communication such as `Zcashd`.
+`Zaino-Fetch` is a library that provides access to the mempool and blockchain data using Zcash's JsonRPC interface. It is primarily used as a backup and for backward compatibility with systems that rely on RPC communication such as `Zcashd`.
 
 ### Functionality
 - RPC Client Implementation:
@@ -153,42 +168,16 @@ Full documentation for `Zaino-State` can be found [here](https://zingolabs.githu
   - Fetches blocks, transactions, and mempool data via RPC.
   - Sends transactions to the network using the `sendrawtransaction` RPC method.
 
+- Block and Transaction Deserialisation logic:
+  - Provides Block and transaction deserialisation implementaions.
+
 - Mempool and CompactFormat access:
   - Provides a simple mempool implementation for use in gRPC service implementations. (This is due to be refactored and possibly moved with the development of `Zaino-State`.)
   - Provides parse implementations for converting "full" blocks and transactions to "compact" blocks and transactions.
 
 - Fallback Mechanism:
   - Acts as a backup when direct access via `zaino-state` is unavailable.
 
-### Interfaces
-- Internal API:
-  - The `jsonrpc::connector` module provides the following JSON-RPC client management functions: `new`, `uri`, `url`, `test_node_and_return_uri`.
-  - The `jsonrpc::connector` module provides the following data retrieval and submission functions: `get_info`, `get_blockchain_info`, `get_address_balance`, `send_raw_transaction`, `get_block`, `get_raw_mempool`, `get_treestate`, `get_subtrees_by_index`, `get_raw_transaction`, `get_address_txids`, `get_address_utxos`. (This may be expanded to match the set of Zcash RPC's that Zaino is taking over from Zcashd.)
-  - The `chain::block` module provides the following block parsing and fetching functions: `get_block_from_node`, `get_nullifiers_from_node`, `FullBlock::parse_from_hex`, `FullBlock::to_compact`, FullBlock::header, FullBlock::transactions, FullBlock::Height, FullBlockHeader::version, FullBlockHeader::hash_prev_block, FullBlockHeader::hash_merkle_root, FullBlockHeader::time, FullBlockHeader::n_bits_bytes, FullBlockHeader::nonce, FullBlockHeader::solution, FullBlockHeader::cached_hash.
-  The `chain::transaction` module provides the following transaction parsing and fetching functions: `FullTransaction::f_overwintered`, `FullTransaction::version`, `FullTransaction::n_version_group_id`, `FullTransaction::consensus_branch_id`, `FullTransaction::transparent_inputs`, `FullTransaction::transparent_outputs`, `FullTransaction::shielded_spends`, `FullTransaction::shielded_outputs`, `FullTransaction::join_splits`, `FullTransaction::orchard_actions`, `FullTransaction::raw_bytes`, `FullTransaction::tx_id`, `FullTransaction::to_compact`.
-  - The `chain::mempool` module provides the following mempool management and fetching functions: `new`, `update`, `get_mempool_txids`, `get_filtered_mempool_txids`, `get_best_block_hash`. (This is due to be refactored and possibly moved with the development of `Zaino-State`.)
-  - Designed to be used by `zaino-serve` transparently.
-
-### Dependencies
-  - `zaino-proto`
-  - `zcash_protocol`
-  - `zebra-chain`
-  - `zebra-rpc`
-  - `tokio`
-  - `tonic`
-  - `http`
-  - `thiserror`
-  - `prost`
-  - `reqwest`
-  - `url`
-  - `serde_json`
-  - `serde`
-  - `hex`
-  - `indexmap`
-  - `base64`
-  - `byteorder`
-  - `sha2`
-
 Full documentation for `Zaino-Fetch` can be found [here](https://zingolabs.github.io/zaino/zaino_fetch/index.html).
 
 
@@ -204,17 +193,6 @@ Full documentation for `Zaino-Fetch` can be found [here](https://zingolabs.githu
   - Uses `prost` to generate Rust types from `.proto` files.
   - Generates client and server stubs for gRPC services.
 
-### Interfaces
-- Generated Code:
-  - Provides Rust modules that can be imported by other crates.
-  - Exposes types and traits required for implementing gRPC services.
-
-### Dependencies
-  - `tonic`
-  - `prost`
-  - `tonic-build`
-  - `which`
-
 * We plan to eventually rely on `LibRustZcash`'s versions but hold our own here for development purposes.
 
 
@@ -224,22 +202,7 @@ The `Zaino-Testutils` and `Integration-Tests` crates are dedicated to testing th
 - `Integration-Tests`: This crate houses integration tests that validate the interaction between different Zaino components and external services like `Zebra` and `Zingolib`.
 
 ### Test Modules
-- `integrations`: Holds Wallet-to-Validator tests that test Zaino's functionality within the compete software stack.
+- `wallet_to_validator`: Holds Wallet-to-Validator tests that test Zaino's functionality within the compete software stack.
 - `client_rpcs`: Holds RPC tests that test the functionality of the LightWallet gRPC services in Zaino and compares the outputs with the corresponding services in `Lightwalletd` to ensure compatibility.
 
-### Dependencies
-  - `zaino-fetch`
-  - `zainod`
-  - `zingolib`
-  - `zaino-testutils`
-  - `zcash_local_net`
-  - `tokio`
-  - `tonic`
-  - `http`
-  - `ctrlc`
-  - `tempfile`
-  - `portpicker`
-  - `tracing-subscriber`
-  - `once_cell`
-
 Full documentation for `Zaino-Testutils` can be found [here](https://zingolabs.github.io/zaino/zaino_testutils/index.html).

docs/rpc_api.md

@@ -1,7 +1,4 @@
 # Zaino RPC APIs
-Zaino's Final RPC API is currently unfinalised and will be completed in this [GitHub Issue](https://github.com/zingolabs/zaino/issues/69).
-
-
 ## Lightwallet gRPC Services
 Zaino Currently Serves the following gRPC services as defined in the [LightWallet Protocol](https://github.com/zcash/librustzcash/blob/main/zcash_client_backend/proto/service.proto):
   - GetLatestBlock (ChainSpec) returns (BlockID)
@@ -26,37 +23,7 @@ Zaino Currently Serves the following gRPC services as defined in the [LightWalle
 
 
 ## Zcash RPC Services
-Zaino has committed to taking over responsibility for serving the following [Zcash RPCs](https://zcash.github.io/rpc/) from Zcashd:
-  - [getaddressbalance](https://zcash.github.io/rpc/getaddressbalance.html)
-  - [getaddressdeltas](https://zcash.github.io/rpc/getaddressdeltas.html)
-  - [getaddressmempool](https://zcash.github.io/rpc/getaddressmempool.html) (**MARKED FOR DEPRECATION**)
-  - [getaddresstxids](https://zcash.github.io/rpc/getaddresstxids.html)
-  - [getaddressutxos](https://zcash.github.io/rpc/getaddressutxos.html)
-  - [getbestblockhash](https://zcash.github.io/rpc/getbestblockhash.html) (**LOW PRIORITY. MARKED FOR POSSIBLE DEPRECATION**)
-  - [getblock](https://zcash.github.io/rpc/getblock.html)
-  - [getblockchaininfo](https://zcash.github.io/rpc/getblockchaininfo.html)
-  - [getblockcount](https://zcash.github.io/rpc/getblockcount.html)
-  - [getblockdeltas](https://zcash.github.io/rpc/getblockdeltas.html)
-  - [getblockhash](https://zcash.github.io/rpc/getblockhash.html)
-  - [getblockhashes](https://zcash.github.io/rpc/getblockhashes.html)
-  - [getblockheader](https://zcash.github.io/rpc/getblockheader.html)
-  - [getchaintips](https://zcash.github.io/rpc/getchaintips.html)
-  - [getdifficulty](https://zcash.github.io/rpc/getdifficulty.html)
-  - [getmempoolinfo](https://zcash.github.io/rpc/getmempoolinfo.html)
-  - [getrawmempool](https://zcash.github.io/rpc/getrawmempool.html)
-  - [getspentinfo](https://zcash.github.io/rpc/getspentinfo.html)
-  - [gettxout](https://zcash.github.io/rpc/gettxout.html)
-  - [gettxoutproof](https://zcash.github.io/rpc/gettxoutproof.html) (**LOW PRIORITY. MARKED FOR POSSIBLE DEPRECATION**)
-  - [gettxoutsetinfo](https://zcash.github.io/rpc/gettxoutsetinfo.html)
-  - [verifytxoutproof](https://zcash.github.io/rpc/verifytxoutproof.html)(**LOW PRIORITY. MARKED FOR POSSIBLE DEPRECATION**)
-  - [z_gettreestate](https://zcash.github.io/rpc/z_gettreestate.html)
-
-Zaino will also provide wrapper functionality for the following RPCs in Zebra (to allow block explorers to fetch all data they require directly from / through Zaino):
-  - [getinfo](https://zcash.github.io/rpc/getinfo.html)
-  - [getmininginfo](https://zcash.github.io/rpc/getmininginfo.html)
-  - [getnetworksolps](https://zcash.github.io/rpc/getnetworksolps.html)
-  - [getnetworkinfo](https://zcash.github.io/rpc/getnetworkinfo.html)
-  - [getpeerinfo](https://zcash.github.io/rpc/getpeerinfo.html)
-  - [ping](https://zcash.github.io/rpc/ping.html)
-
+Zaino has also committed to taking over responsibility for serving all [Zcash RPC Services](https://zcash.github.io/rpc/) required by non-validator (miner) clients from Zcashd.
+A full specification of the Zcash RPC services served by Zaino, and their current state of development, can be seen [here](./Zaino-zcash-rpcs.pdf).
 
+