rpc benchmark init: directly read DB based on schema #20378

gegaowp · 2024-11-21T23:00:10Z

Description

main things in this pr

generate template queries based on tables in the DB, excluding __diesel_schema_migrations
enrich the template with values read from DB and execute queries in parallel
configs of concurrency and duration time
metrics collection and reports

Test plan

local run with a local DB populated with latest sui-indexer-alt-schema

cargo run --bin sui-rpc-benchmark -- direct \
    --db-url "postgres://postgres:postgres@localhost/gegao" \
    --concurrency 10 \
    --duration-secs 10

report with local DB


Total queries: 211772
Total errors: 0
Average latency: 0.46ms

Per-table statistics:
  obj_info                       queries: 33526    errors: 0        avg latency: 0.49ms
  ev_struct_inst                 queries: 31598    errors: 0        avg latency: 0.47ms
  tx_calls                       queries: 28752    errors: 0        avg latency: 0.45ms
  ev_emit_mod                    queries: 18321    errors: 0        avg latency: 0.44ms
  tx_affected_objects            queries: 11495    errors: 0        avg latency: 0.43ms
  tx_affected_addresses          queries: 11472    errors: 0        avg latency: 0.43ms
  sum_packages                   queries: 10759    errors: 0        avg latency: 0.55ms
  tx_kinds                       queries: 8140     errors: 0        avg latency: 0.43ms
  coin_balance_buckets           queries: 7542     errors: 0        avg latency: 0.45ms
  obj_versions                   queries: 7485     errors: 0        avg latency: 0.43ms
  kv_epoch_ends                  queries: 3750     errors: 0        avg latency: 0.45ms
  watermarks                     queries: 3742     errors: 0        avg latency: 0.44ms
  tx_digests                     queries: 3690     errors: 0        avg latency: 0.42ms
  tx_balance_changes             queries: 3662     errors: 0        avg latency: 0.42ms
  kv_checkpoints                 queries: 3619     errors: 0        avg latency: 0.43ms
  cp_sequence_numbers            queries: 3590     errors: 0        avg latency: 0.42ms
  kv_transactions                queries: 3463     errors: 0        avg latency: 0.43ms
  kv_protocol_configs            queries: 3451     errors: 0        avg latency: 0.44ms
  kv_epoch_starts                queries: 3448     errors: 0        avg latency: 0.71ms
  kv_genesis                     queries: 3424     errors: 0        avg latency: 0.42ms
  kv_objects                     queries: 3422     errors: 0        avg latency: 0.42ms
  kv_feature_flags               queries: 3421     errors: 0        avg latency: 0.42ms

Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

vercel · 2024-11-21T23:00:17Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
sui-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 18, 2025 6:44pm

3 Skipped Deployments

Name	Status	Preview	Updated (UTC)
multisig-toolkit	⬜️ Ignored (Inspect)	Visit Preview	Feb 18, 2025 6:44pm
sui-kiosk	⬜️ Ignored (Inspect)	Visit Preview	Feb 18, 2025 6:44pm
sui-typescript-docs	⬜️ Ignored (Inspect)	Visit Preview	Feb 18, 2025 6:44pm

lxfind · 2024-11-25T20:24:33Z

generate template queries based on schema, specifically table primary key and indexes

What is the rational behind these generated queries? Would they be representative to what we are interested in?

gegaowp · 2024-11-25T20:33:10Z

@lxfind the generated queries now are basically "all supported queries based on primary key and indices" and does not consider the representativeness, I plan make it configurable for example taking in a file of weights to make it representative as a followup.

crates/sui-rpc-benchmark/src/direct/query_generator.rs

crates/sui-rpc-benchmark/src/direct/query_executor.rs

amnn

👍

amnn · 2025-02-14T19:38:05Z

crates/sui-rpc-benchmark/src/lib.rs

+            long,
+            default_value = "postgres://postgres:postgres@localhost:5432/sui"
+        )]
+        db_url: String,


The idea was to use Url as the field type here so that clap will handle the parsing validation errors for you -- you can still supply a default value using default_value_t, I think.

amnn · 2025-02-14T19:44:18Z

crates/sui-rpc-benchmark/src/direct/query_enricher.rs

+    ) -> Result<Vec<EnrichedBenchmarkQuery>> {
+        let mut enriched_queries = Vec::new();
+        for query in queries {
+            let enriched = self.enrich_query(&query).await?;


Do this concurrently?

crates/sui-rpc-benchmark/src/direct/query_executor.rs

amnn · 2025-02-14T19:46:38Z

crates/sui-rpc-benchmark/src/direct/query_executor.rs

+        while Instant::now() < deadline {
+            let enriched = enriched_queries
+                .choose(&mut query_rng)
+                .ok_or_else(|| anyhow::anyhow!("No queries available"))?;


Suggested change

.ok_or_else(|| anyhow::anyhow!("No queries available"))?;

.context("No queries available")?;

it's an option so no original Result to add additional context to?

Context works on Option too (to convert it into an anyhow::Result).

yes, I saw your PR and will amend this change to the latest commit, TIL, thanks!

…d executor

gegaowp requested a review from ronny-mysten as a code owner November 21, 2024 23:00

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env November 21, 2024 23:00 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs November 21, 2024 23:01 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 9c98761 to 1069b2c Compare November 21, 2024 23:08

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env November 21, 2024 23:08 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs November 21, 2024 23:09 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 1069b2c to 8b0818c Compare November 21, 2024 23:10

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env November 21, 2024 23:10 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs November 21, 2024 23:11 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 8b0818c to eb96893 Compare November 21, 2024 23:12

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env November 21, 2024 23:12 — with GitHub Actions Inactive

gegaowp requested review from amnn, lxfind, bmwill, emmazzz and wlmyng November 21, 2024 23:14

vercel bot deployed to Preview – sui-docs November 21, 2024 23:14 View deployment

gegaowp changed the title ~~rpc benchmark init: direct reading DB based on schema~~ rpc benchmark init: directly read DB based on schema Nov 21, 2024

lxfind reviewed Nov 25, 2024

View reviewed changes

crates/sui-rpc-benchmark/src/direct/query_generator.rs Outdated Show resolved Hide resolved

crates/sui-rpc-benchmark/src/direct/query_generator.rs Outdated Show resolved Hide resolved

crates/sui-rpc-benchmark/src/direct/query_executor.rs Outdated Show resolved Hide resolved

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env November 26, 2024 21:05 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs November 26, 2024 21:06 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 902755b to 43ab64b Compare January 8, 2025 10:17

gegaowp requested a review from lxfind January 8, 2025 10:18

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env January 8, 2025 10:18 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs January 8, 2025 10:19 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 43ab64b to 47dcc16 Compare January 8, 2025 10:26

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env January 8, 2025 10:26 — with GitHub Actions Inactive

gegaowp force-pushed the rpc-benchmark-init branch from d6648cc to 592ae3d Compare February 3, 2025 22:25

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 3, 2025 22:25 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 3, 2025 22:27 View deployment

amnn approved these changes Feb 14, 2025

View reviewed changes

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 18, 2025 16:42 — with GitHub Actions Inactive

gegaowp enabled auto-merge (squash) February 18, 2025 16:42

vercel bot deployed to Preview – sui-docs February 18, 2025 16:47 View deployment

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 18, 2025 16:51 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 18, 2025 16:53 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from ac9a7eb to 2d14e5e Compare February 18, 2025 17:29

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 18, 2025 17:29 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 18, 2025 17:30 View deployment

gegaowp force-pushed the rpc-benchmark-init branch from 2d14e5e to eb222aa Compare February 18, 2025 18:06

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 18, 2025 18:07 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 18, 2025 18:08 View deployment

gegaowp added 10 commits February 18, 2025 13:40

rpc benchmark: init commit of direct reading DB based on schema

de928ba

address comments: migration path as arg and return values

34711b3

refactor, add metrics and parallel run

5d6b1e9

address comments

65dc5db

comments

5fe8678

comments, including rafactor to split template generator, enricher an…

50f6181

…d executor

concurrent query enrich

f570d2f

one rng

09b27fa

typed Url in clap cmd

3306177

bb8 versions

c3519ba

gegaowp force-pushed the rpc-benchmark-init branch from eb222aa to c3519ba Compare February 18, 2025 18:42

gegaowp temporarily deployed to sui-typescript-aws-kms-test-env February 18, 2025 18:42 — with GitHub Actions Inactive

vercel bot deployed to Preview – sui-docs February 18, 2025 18:44 View deployment

gegaowp merged commit 7aaab9c into MystenLabs:main Feb 18, 2025
43 checks passed

gegaowp deleted the rpc-benchmark-init branch February 18, 2025 19:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc benchmark init: directly read DB based on schema #20378

rpc benchmark init: directly read DB based on schema #20378

gegaowp commented Nov 21, 2024 •

edited

Loading

vercel bot commented Nov 21, 2024 •

edited

Loading

lxfind commented Nov 25, 2024

gegaowp commented Nov 25, 2024

amnn left a comment

amnn Feb 14, 2025

amnn Feb 14, 2025

amnn Feb 14, 2025

gegaowp Feb 18, 2025

amnn Feb 21, 2025

gegaowp Feb 21, 2025

	.ok_or_else(\|\| anyhow::anyhow!("No queries available"))?;
	.context("No queries available")?;

rpc benchmark init: directly read DB based on schema #20378

rpc benchmark init: directly read DB based on schema #20378

Conversation

gegaowp commented Nov 21, 2024 • edited Loading

Description

Test plan

Release notes

vercel bot commented Nov 21, 2024 • edited Loading

lxfind commented Nov 25, 2024

gegaowp commented Nov 25, 2024

amnn left a comment

Choose a reason for hiding this comment

amnn Feb 14, 2025

Choose a reason for hiding this comment

amnn Feb 14, 2025

Choose a reason for hiding this comment

amnn Feb 14, 2025

Choose a reason for hiding this comment

gegaowp Feb 18, 2025

Choose a reason for hiding this comment

amnn Feb 21, 2025

Choose a reason for hiding this comment

gegaowp Feb 21, 2025

Choose a reason for hiding this comment

gegaowp commented Nov 21, 2024 •

edited

Loading

vercel bot commented Nov 21, 2024 •

edited

Loading