Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

indexer(visitor): avoid fully deserializing dynamic field #19520

Merged
merged 1 commit into from
Sep 30, 2024

Conversation

amnn
Copy link
Contributor

@amnn amnn commented Sep 24, 2024

Description

Use FieldVisitor to extract necessary information from a dynamic field object, rather than fully expanding it, which can fail if the overall size is too large.

Unfortunately, because DynamicFieldInfo includes a structured representation of the field name, this operation can still fail if the name alone is too large to deserialize using BoundedVisitor, but we can at least avoid deserializing the value.

Test plan

Build and run the indexer reader against a mainnet DB, and check for 0x5:

sui$ cargo run --bin sui-indexer -- --database-url "$DB" --pool-size 10 \
  json-rpc-service --rpc-client-url "https://fullnode.mainnet.sui.io:443"

Make the request:

$ curl -LX POST  "https:/fullnode.mainnet.sui.io:443" \
        --header 'Content-Type: application/json' \
        --data-raw '{
            "jsonrpc": "2.0",
            "method": "suix_getDynamicFields",
            "id": 1,
            "params": ["0x5"]
        }' | jq  -C . | less -r

Expects a response like:

{
  "jsonrpc": "2.0",
  "result": {
    "data": [
      {
        "name": {
          "type": "u64",
          "value": "2"
        },
        "bcsName": "LQM2cdzDY3",
        "type": "DynamicField",
        "objectType": "0x3::sui_system_state_inner::SuiSystemStateInnerV2",
        "objectId": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
        "version": 339253281,
        "digest": "89nPMzp31fiSy39a7fg6TBzALdm3xA4Byd4hWr2QLahg"
      }
    ],
    "nextCursor": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
    "hasNextPage": false
  },
  "id": 1
}

Stack


Release notes

Check each box that your changes affect. If none of the boxes relate to your changes, release notes aren't required.

For each box you select, include information after the relevant heading that describes the impact of your changes that a user might notice and any actions they must take to implement updates.

  • Protocol:
  • Nodes (Validators and Full nodes):
  • Indexer:
  • JSON-RPC:
  • GraphQL:
  • CLI:
  • Rust SDK:
  • REST API:

@amnn amnn requested review from tzakian, bmwill and gegaowp September 24, 2024 15:18
@amnn amnn self-assigned this Sep 24, 2024
Copy link

vercel bot commented Sep 24, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
sui-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Sep 30, 2024 11:09pm
3 Skipped Deployments
Name Status Preview Comments Updated (UTC)
multisig-toolkit ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 11:09pm
sui-kiosk ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 11:09pm
sui-typescript-docs ⬜️ Ignored (Inspect) Visit Preview Sep 30, 2024 11:09pm

Copy link
Contributor

@gegaowp gegaowp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good move! btw I plan to further simplify codes around as now most meat of df_info are not longer needed.

@amnn
Copy link
Contributor Author

amnn commented Sep 25, 2024

@gegaowp -- I noticed that -- this needs to be simplified for FieldVisitor to be useful on the writer side -- I was planning on doing that anyway -- probably tomorrow, but if you get to it before me, all good!

@gegaowp
Copy link
Contributor

gegaowp commented Sep 25, 2024

@gegaowp -- I noticed that -- this needs to be simplified for FieldVisitor to be useful on the writer side -- I was planning on doing that anyway -- probably tomorrow, but if you get to it before me, all good!

@amnn awesome and go ahead please, will review when you have it :)

@amnn amnn force-pushed the amnn/gql-df-visitor branch from eb49b74 to b11004c Compare September 30, 2024 22:49
@amnn amnn force-pushed the amnn/idx-reader-df-visitor branch from 2fa3ab4 to ed6d3da Compare September 30, 2024 22:49
Base automatically changed from amnn/gql-df-visitor to amnn/mv-visitor-refactors September 30, 2024 23:01
## Description

Use `FieldVisitor` to extract necessary information from a dynamic field
object, rather than fully expanding it, which can fail if the overall
size is too large.

Unfortunately, because `DynamicFieldInfo` includes a structured
representation of the field name, this operation can still fail if the
name alone is too large to deserialize using `BoundedVisitor`, but we
can at least avoid deserializing the value.

## Test plan

Build and run the indexer reader against a mainnet DB, and check for
`0x5`:

```
sui$ cargo run --bin sui-indexer -- --database-url "$DB" --pool-size 10 \
  json-rpc-service --rpc-client-url "https://fullnode.mainnet.sui.io:443"
```

Make the request:
```
$ curl -LX POST  "https:/fullnode.mainnet.sui.io:443" \
        --header 'Content-Type: application/json' \
        --data-raw '{
            "jsonrpc": "2.0",
            "method": "suix_getDynamicFields",
            "id": 1,
            "params": ["0x5"]
        }' | jq  -C . | less -r
```

Expects a response like:
```
{
  "jsonrpc": "2.0",
  "result": {
    "data": [
      {
        "name": {
          "type": "u64",
          "value": "2"
        },
        "bcsName": "LQM2cdzDY3",
        "type": "DynamicField",
        "objectType": "0x3::sui_system_state_inner::SuiSystemStateInnerV2",
        "objectId": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
        "version": 339253281,
        "digest": "89nPMzp31fiSy39a7fg6TBzALdm3xA4Byd4hWr2QLahg"
      }
    ],
    "nextCursor": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
    "hasNextPage": false
  },
  "id": 1
}
```
@amnn amnn force-pushed the amnn/idx-reader-df-visitor branch from ed6d3da to dc89159 Compare September 30, 2024 23:03
@amnn amnn merged commit 0557758 into amnn/mv-visitor-refactors Sep 30, 2024
38 of 45 checks passed
@amnn amnn deleted the amnn/idx-reader-df-visitor branch September 30, 2024 23:04
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Make it so that deserializing dynamic field info does not inflate the
whole field -- just the outer struct -- which in turn means that this
operation cannot fail.

## Test plan

```
sui$ cargo simtest
sui-adapter-transactional-tests$ cargo simtest
```

## Stack

- #19517 
- #19518 
- #19519 
- #19520 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Make it so that deserializing dynamic field info does not inflate the
whole field -- just the outer struct -- which in turn means that this
operation cannot fail.

## Test plan

```
sui-core$ cargo simtest
sui-adapter-transactional-tests$ cargo simtest
```

## Stack

- #19517 
- #19518 
- #19519 
- #19520 
- #19521 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Sep 30, 2024
…th (#19565)

## Description

Use `FieldVisitor` to extract the dynamic field kind from a serialized
MoveObject without deserializing the whole thing. This makes it so that
the indexing process cannot fail if presented with an overly large
dynamic field.

This PR also simplifies dynamic field indexing, taking advantage of the
fact that we now only store the `df_kind`, and not any other part of
`df_info`.

## Test plan

```
sui-indexer$ cargo nextest run
sui-graphql-e2e-tests$ cargo nextest run
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Replace the legacy implementations of `get_all_uids` with the one that
has been used in `latest` for some time now. This version avoids
inflating the whole struct just to read its UIDs.

The main motivation in making the switch is to reduce the number of
call-sites that use `simple_deserialize`, so that we can eventually get
rid of it entirely, and replace it with a visitor based ipmplementation
that includes the appropriate caveats around inflating a whole struct in
memory.

## Test plan

```
sui$ cargo simtest
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Use a custom annotated visitor to implement detecting wrapped objects in
test scenario. Unlike other instances where we have switched to a custom
visitor, there isn't an OOM risk or risk of failure at this call site,
because it is only used in Move tests, which are only run locally, but
the motivation for this change is to avoid a reliance on
`simple_deserialize` in the codebase, as it is too easy to copy this
pattern and introduce a vulnerability elsewhere.

Eventually `simple_deserialize` will go away and be replaced by
something that is based on the new annotated visitor, but looks scary
enough to use that people think twice before reaching for it.

## Test plan

```
sui$ cargo nextest run -p sui-framework-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Further simplification to dynamic field indexing, where we avoid
deserializing dynamic fields altogether and extract the only piece of
information we need (whether the DF is a dynamic field or a dynamic
object field) from its type.

DF deserialization was also the only reason to perform package
resolution during indexing, so this change also means we can remove the
package resolver, package buffer and logic to keep those up-to-date
(pushing in new packages before indexing a checkpoint, and "evicting"
packages after a checkpoint had been processed).

## Test plan

```
sui$ cargo nextest run -p sui-graphql-e2e-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 
- #19621 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Sep 30, 2024
…th (#19565)

## Description

Use `FieldVisitor` to extract the dynamic field kind from a serialized
MoveObject without deserializing the whole thing. This makes it so that
the indexing process cannot fail if presented with an overly large
dynamic field.

This PR also simplifies dynamic field indexing, taking advantage of the
fact that we now only store the `df_kind`, and not any other part of
`df_info`.

## Test plan

```
sui-indexer$ cargo nextest run
sui-graphql-e2e-tests$ cargo nextest run
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Replace the legacy implementations of `get_all_uids` with the one that
has been used in `latest` for some time now. This version avoids
inflating the whole struct just to read its UIDs.

The main motivation in making the switch is to reduce the number of
call-sites that use `simple_deserialize`, so that we can eventually get
rid of it entirely, and replace it with a visitor based ipmplementation
that includes the appropriate caveats around inflating a whole struct in
memory.

## Test plan

```
sui$ cargo simtest
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Use a custom annotated visitor to implement detecting wrapped objects in
test scenario. Unlike other instances where we have switched to a custom
visitor, there isn't an OOM risk or risk of failure at this call site,
because it is only used in Move tests, which are only run locally, but
the motivation for this change is to avoid a reliance on
`simple_deserialize` in the codebase, as it is too easy to copy this
pattern and introduce a vulnerability elsewhere.

Eventually `simple_deserialize` will go away and be replaced by
something that is based on the new annotated visitor, but looks scary
enough to use that people think twice before reaching for it.

## Test plan

```
sui$ cargo nextest run -p sui-framework-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Sep 30, 2024
## Description

Further simplification to dynamic field indexing, where we avoid
deserializing dynamic fields altogether and extract the only piece of
information we need (whether the DF is a dynamic field or a dynamic
object field) from its type.

DF deserialization was also the only reason to perform package
resolution during indexing, so this change also means we can remove the
package resolver, package buffer and logic to keep those up-to-date
(pushing in new packages before indexing a checkpoint, and "evicting"
packages after a checkpoint had been processed).

## Test plan

```
sui$ cargo nextest run -p sui-graphql-e2e-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 
- #19621 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Use `FieldVisitor` to extract necessary information from a dynamic field
object, rather than fully expanding it, which can fail if the overall
size is too large.

Unfortunately, because `DynamicFieldInfo` includes a structured
representation of the field name, this operation can still fail if the
name alone is too large to deserialize using `BoundedVisitor`, but we
can at least avoid deserializing the value.

## Test plan

Build and run the indexer reader against a mainnet DB, and check for
`0x5`:

```
sui$ cargo run --bin sui-indexer -- --database-url "$DB" --pool-size 10 \
  json-rpc-service --rpc-client-url "https://fullnode.mainnet.sui.io:443"
```

Make the request:
```
$ curl -LX POST  "https:/fullnode.mainnet.sui.io:443" \
        --header 'Content-Type: application/json' \
        --data-raw '{
            "jsonrpc": "2.0",
            "method": "suix_getDynamicFields",
            "id": 1,
            "params": ["0x5"]
        }' | jq  -C . | less -r
```

Expects a response like:
```
{
  "jsonrpc": "2.0",
  "result": {
    "data": [
      {
        "name": {
          "type": "u64",
          "value": "2"
        },
        "bcsName": "LQM2cdzDY3",
        "type": "DynamicField",
        "objectType": "0x3::sui_system_state_inner::SuiSystemStateInnerV2",
        "objectId": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
        "version": 339253281,
        "digest": "89nPMzp31fiSy39a7fg6TBzALdm3xA4Byd4hWr2QLahg"
      }
    ],
    "nextCursor": "0x5b890eaf2abcfa2ab90b77b8e6f3d5d8609586c3e583baf3dccd5af17edf48d1",
    "hasNextPage": false
  },
  "id": 1
}
```

## Stack

- #19517 
- #19518 
- #19519 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Make it so that deserializing dynamic field info does not inflate the
whole field -- just the outer struct -- which in turn means that this
operation cannot fail.

## Test plan

```
sui$ cargo simtest
sui-adapter-transactional-tests$ cargo simtest
```

## Stack

- #19517 
- #19518 
- #19519 
- #19520 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Make it so that deserializing dynamic field info does not inflate the
whole field -- just the outer struct -- which in turn means that this
operation cannot fail.

## Test plan

```
sui-core$ cargo simtest
sui-adapter-transactional-tests$ cargo simtest
```

## Stack

- #19517 
- #19518 
- #19519 
- #19520 
- #19521 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
amnn added a commit that referenced this pull request Oct 1, 2024
…th (#19565)

## Description

Use `FieldVisitor` to extract the dynamic field kind from a serialized
MoveObject without deserializing the whole thing. This makes it so that
the indexing process cannot fail if presented with an overly large
dynamic field.

This PR also simplifies dynamic field indexing, taking advantage of the
fact that we now only store the `df_kind`, and not any other part of
`df_info`.

## Test plan

```
sui-indexer$ cargo nextest run
sui-graphql-e2e-tests$ cargo nextest run
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Replace the legacy implementations of `get_all_uids` with the one that
has been used in `latest` for some time now. This version avoids
inflating the whole struct just to read its UIDs.

The main motivation in making the switch is to reduce the number of
call-sites that use `simple_deserialize`, so that we can eventually get
rid of it entirely, and replace it with a visitor based ipmplementation
that includes the appropriate caveats around inflating a whole struct in
memory.

## Test plan

```
sui$ cargo simtest
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Use a custom annotated visitor to implement detecting wrapped objects in
test scenario. Unlike other instances where we have switched to a custom
visitor, there isn't an OOM risk or risk of failure at this call site,
because it is only used in Move tests, which are only run locally, but
the motivation for this change is to avoid a reliance on
`simple_deserialize` in the codebase, as it is too easy to copy this
pattern and introduce a vulnerability elsewhere.

Eventually `simple_deserialize` will go away and be replaced by
something that is based on the new annotated visitor, but looks scary
enough to use that people think twice before reaching for it.

## Test plan

```
sui$ cargo nextest run -p sui-framework-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:

---------

Co-authored-by: Timothy Zakian <tim@mystenlabs.com>
amnn added a commit that referenced this pull request Oct 1, 2024
## Description

Further simplification to dynamic field indexing, where we avoid
deserializing dynamic fields altogether and extract the only piece of
information we need (whether the DF is a dynamic field or a dynamic
object field) from its type.

DF deserialization was also the only reason to perform package
resolution during indexing, so this change also means we can remove the
package resolver, package buffer and logic to keep those up-to-date
(pushing in new packages before indexing a checkpoint, and "evicting"
packages after a checkpoint had been processed).

## Test plan

```
sui$ cargo nextest run -p sui-graphql-e2e-tests
```

## Stack

- #19517
- #19518 
- #19519 
- #19520 
- #19521 
- #19548 
- #19554
- #19565 
- #19576 
- #19621 

---

## Release notes

Check each box that your changes affect. If none of the boxes relate to
your changes, release notes aren't required.

For each box you select, include information after the relevant heading
that describes the impact of your changes that a user might notice and
any actions they must take to implement updates.

- [ ] Protocol: 
- [ ] Nodes (Validators and Full nodes): 
- [ ] Indexer: 
- [ ] JSON-RPC: 
- [ ] GraphQL: 
- [ ] CLI: 
- [ ] Rust SDK:
- [ ] REST API:
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants