Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: bump all major dependencies #2007

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

SuperFluffy
Copy link
Member

@SuperFluffy SuperFluffy commented Feb 28, 2025

Summary

Bump all major dependencies

Background

An update to our Astria's dependencies was long overdue. Notable developments have been:

  1. penumbra releasing most of their core crates to crates.io (changing the penumbra-* prefix to penumbra-sdk-*).
  2. http and hyper getting stable 1.0 releases
  3. celestia-tendermint being replaced by upstream tendermint
  4. tendermint having seen a 0.40.0 release with several fixes (TODO: @joroshiba you pointed out some relevant patches we need?)

Changes

major dependency upgrades:

  1. axum: 0.7.9
  2. celestia-types: 0.10.0
  3. cnidarium: 0.83.0 (instead of upstream rev ac7abacc)
  4. http: 1.2.0
  5. hyper: 1.6.0
  6. ibc-types: 0.15.0
  7. jsonrpsee: 0.24.8
  8. metrics: 0.24.1
  9. metrics-exporter-prometheus: 0.16.2
  10. opentelemetry: 0.27.1
  11. opentelemetry-otlp: 0.27.0
  12. opentelemetry-semantic-conventions: 0.27.0
  13. opentelemetry_sdk: 0.27.1
  14. pbjson-build: 0.7.0
  15. pbjson-types: 0.7.0
  16. penumbra-ibc (renamed to penumbra-sdk-ibc): 1.0.1
  17. penumbra-proto (renamed to penumbra-sdk-proto): 1.0.1
  18. penumbra-tower-trace (renamed to penumbra-sdk-tower-trace): 1.0.1
  19. prost: 0.13.4
  20. prost-build: 0.13.4
  21. reqwest: 0.12.12
  22. tendermint: 0.40.1
  23. tendermint-config: 0.40.1
  24. tendermint-proto: 0.40.1
  25. tendermint-rpc: 0.40.1
  26. tonic: 0.12.3
  27. tonic-build: 0.12.3
  28. tower: 0.5.2
  29. tower-http: 0.6.2
  30. tracing-opentelemetry: 0.28.0

removed dependencies

  1. celestia-tendermint: celestia-types
    now relies on upstream tendermint and
    celestia-tendermint is no longer maintained.
  2. opentelemetry-stdout: not to be used in
    production code and the APIs we used were removed.
  3. isahc: replaced by reqwest because it has not
    seen updates in a long time and was not updated
    to the work with hyper and http 1.0 stabilization.
  4. http-body and http-body-util (not needed; can use tonic reexports).

notable overrides:

this patch relies on changes to a number of
dependencies that have not yet been upstreamed:

  1. penumbra-sdk-ibc, penumbra-sdk-proto,
    penumbra-sdk-tower-trace need a bump to their
    dependencies on tonic, tower, tower-actor. Tracked
    in
    bump tower@v0.5.2, tower-abci@v0.19.1, tower-actor@v0.2.0 penumbra-zone/penumbra#5055
  2. tower-actor: needs a bump to tower. Tracked in
    prepare v0.2.0, bump tower@v0.5.2, quiet warnings and minimize tokio penumbra-zone/tower-actor#5
  3. tonic: has its tower dependency bumped to
    0.5.2 upstream but is still awaiting a release to
    0.13.0. The tower patch was merged in
    hyperium/tonic@58345a6

Other and crate/service specific changes

Several crates that had a custom defined
dependency now use the workspace definition.

All services lost their <SERVICVE_NAME>_PRETTY_PRINT
env var, because it was no longer necessary. See the
telemetry changes below.

Conductor

celestia-types no longer has celestia-tendermint
as a dependency and instead uses tendermint 0.40
(the same as Astria uses). This required changing
a bunch imports and type paths.

Blob::new now requires an AppVersion argument
(only needed in tests. Mainnet is running v3.2.0,
so the blackbox tests sets the same, but the
conductor business logic is oblivious to that).

Type celestia_types::ataAvailabilityHeader no longer
has public fields and instead provides a constructor
(only in tests, Conductor business logic is again oblivious).

From celestia-node v0.15.0 blob.GetAll no longer
returns an error if a single requested namespace
(out of a range of namespaces) does not contain
blobs. Instead, the namespace does not show up in
the list of blobs. If there are no blobs at all
then the RPC omits the field entirely, which
the Rust celestia-rpc crate represents as a
Option<Vec<Blob>>. All code related to
mapping errors like BlobNotFound to an empty
vec have been removed. The current version of
celestia-node on Astria's mainnet is 0.20.0.

tower introduced the BoxCloneSyncService, which
makes constructing a rate-limited, buffered
service a bit easier.

notable sequencer-client changes:

changes except for tests worked out of the box,
but tokio for some reason was a direct dependency.
Changed it to derive from the workspace.

tendermint_rpc::endpoint::broadcast::Response
has a new field codespace, which needed to be
populated in test code.

Composer

the api server needed refactoring because axum
had breaking changes. To avoid dealing with
the type parameters in the new axum::serve::Serve
future, composer turns it into a trait object and
wraps it in a api::Serve future.

The geth related blackbox tests also started
to be more flaky for unknown reasons - extending
the timeout duration for the tests mostly fixes
the flakiness. But this will have to be
be investigated in a follow-up PR.

Bridge-Withdrawer

axum::Server was replaced by axum::serve::Serve,
which requires a tokio::net::TcpListener to be
instantiated. We have introduced api::Serve as a
type-erased wrapper to axum::serve::Serve to
avoid having to specify its type parameteres while
still providing access to its bound socket.
However, because of these changes
BridgeWithdrawer::new is now async.

the http dependency is now specified through the
workspace.

The shutdown signal for the API server was also
changed to be a CancellationToken.

Sequencer-Relayer

axum underwent heavy changes and has replaced
axum::Server by axum::serve::Serve future. This
patch wraps the future to avoid having to name its
type, erasing it, while still giving access to the
bound tcp socket address. Because its constructor
takes a tokio::net::TcpListener, the API server
setup is now async, which in turn required
SequencerRelayer::new to become async.
Due to type erasure + wrapping, the shutdown
signal closure now has to be passed in early on
instead of right before entering
sequencer-relayer's runloop. While doing so,
the oneshot channel was replaced by a cancellation
token.

Because Celestia blobs now require an app version,
the relayer write path now sets it to version 3
(because astria mainnet is targetting celestia-app
3.2.0, and the consensus node is targetting
3.3.0).

In tests, isahc was replaced by reqwest. The
entire workspace is migrating to hyper 1+ and
http 1+ and isahc has not seen upgrades in several
months and there is no work underway to make it
compatible with http 1+.

dependency tower was removed because it was not
used.

Auctioneer

there was no need for the direct dependencies
on http-body and http-body-util because tonic
already provided reexports and/or type aliases.

Telemetry

opentelemetry had a couple of breaking changes
that required reworking how the otel pipeline
is set up.

the most visible change is opentelemetry-stdout
no longer allowing to write to stderr or another
sink. because the otel stdout writer was never
used and because opentelemetry-stdout itself
notes that is should not be used in production
contexts this patch removes it entirely (and
with it the the need to explicitly set pretty
printed logs; if services are connected to a
TTY or the FORCE_STDOUT flag is set, they will
always write to stdout).

Testing

Only dependencies were upgraded. If tests and smoke tests still work then nothing should be broken.

Changelogs

Ensure all relevant changelog files are updated as necessary. See
keepachangelog for change
categories. Replace this text with e.g. "Changelogs updated." or "No updates
required." to acknowledge changelogs have been considered.

Related Issues

Link any issues that are related, prefer full GitHub links.

closes

@github-actions github-actions bot added conductor pertaining to the astria-conductor crate proto pertaining to the Astria Protobuf spec sequencer pertaining to the astria-sequencer crate sequencer-relayer pertaining to the astria-sequencer-relayer crate composer pertaining to composer cd labels Feb 28, 2025
@SuperFluffy SuperFluffy force-pushed the superfluffy/tendermint-0.40 branch from bfebff2 to bbe36c1 Compare February 28, 2025 17:20
major bumps are:

1. axum: 0.7.9
2. celestia-types: 0.10.0
3. cnidarium: 0.83.0 (instead of upstream rev ac7abacc)
4. http: 1.2.0
5. hyper: 1.6.0
6. ibc-types: 0.15.0
7. jsonrpsee: 0.24.8
8. metrics: 0.24.1
9. metrics-exporter-prometheus: 0.16.2
10. opentelemetry: 0.27.1
11. opentelemetry-otlp: 0.27.0
12. opentelemetry-semantic-conventions: 0.27.0
13. opentelemetry_sdk: 0.27.1
14. pbjson-build: 0.7.0
15. pbjson-types: 0.7.0
16. penumbra-ibc (renamed to penumbra-sdk-ibc): 1.0.1
16. penumbra-proto (renamed to penumbra-sdk-proto): 1.0.1
17. penumbra-tower-trace (renamed to penumbra-sdk-tower-trace): 1.0.1
18. prost: 0.13.4
19. prost-build: 0.13.4
20. reqwest: 0.12.12
21. tendermint: 0.40.1
22. tendermint-config: 0.40.1
23. tendermint-proto: 0.40.1
24. tendermint-rpc: 0.40.1
25. tonic: 0.12.3
26. tonic-build: 0.12.3
27. tower: 0.5.2
28. tower-http: 0.6.2
29. tracing-opentelemetry: 0.28.0

removed:

1. celestia-tendermint: celestia-types
now relies on upstream tendermint and
celestia-tendermint is no longer maintained.
2. opentelemetry-stdout: not to be used in
production code and the APIs we used were removed.
3. isahc: replaced by reqwest because it has not
seen updates in a long time and was not updated
to the work with hyper and http 1.0 stabilization.

notable overrides:

this patch relies on changes to a number of
dependencies that have not yet been upstreamed:

1. penumbra-sdk-ibc, penumbra-sdk-proto,
penumbra-sdk-tower-trace need a bump to their
dependencies on tonic, tower, tower-actor. Tracked
in
penumbra-zone/penumbra#5055
2. tower-actor: needs a bump to tower. Tracked in
penumbra-zone/tower-actor#5
3. tonic: has its tower dependency bumped to
0.5.2 upstream but is still awaiting a release to
0.13.0. The tower patch was merged in
hyperium/tonic@58345a6

Several crates that had a custom defined
dependency now use the workspace definition.

notable conductor changes:

upgrade the following crates:

celestia-types no longer has celestia-tendermint
as a dependency and instead uses tendermint 0.40
(the same as Astria uses). This required changing
a bunch imports and type paths.

`Blob::new` now requires an `AppVersion` argument
(only needed in tests. Mainnet is running v3.2.0,
so the blackbox tests sets the same, but the
conductor business logic is oblivious to that).

The DataAvailability no longer has public fields
and instead provides a constructor (only in tests,
Conductor business logic is again oblivious).

From celestia-node v0.15.0 blob.GetAll no longer
returns an error if a single requested namespace
(out of a range of namespaces) does not contain
blobs. Instead, the namespace does not show up in
the list of blobs. If there are no blobs at all
then the RPC omits the field entirely, which
the Rust celestia-rpc crate represents as a
`Option<Vec<Blob>>`. All code related to
mapping errors like `BlobNotFound` to an empty
vec have been removed. The current version of
celestia-node on Astria's mainnet is 0.20.0.

tower introduced the BoxCloneSyncService, which
makes constructing a rate-limited, buffered
service a bit easier.

notable sequencer-client changes:

changes except for tests worked out of the box,
but tokio for some reason was a direct dependency.
Changed it to derive from the workspace.

tendermint_rpc::endpoint::broadcast::Response
has a new field codespace, which needed to be
populated in test code.

noteable composer changes:

the api server needed refactoring because axum
had breaking changes. To avoid dealing with
the type parameters in the new axum::serve::Serve
future, composer turns it into a trait object and
wraps it in a api::Serve future.

The geth related blackbox tests also started
to be more flaky for unknown reasons - extending
the timeout duration for the tests mostly fixes
the flakiness. But this will have to be
be investigated in a follow-up PR.

notable bridge-withdrawer changes:

axum::Server was replaced by axum::serve::Serve,
which requires a tokio::net::TcpListener to be
instantiated. We have introduced `api::Serve` as a
type-erased wrapper to `axum::serve::Serve` to
avoid having to specify its type parameteres while
still providing access to its bound socket.
However, because of these changes
BridgeWithdrawer::new is now async.

the http dependency is now specified through the
workspace.

The shutdown signal for the API server was also
changed to be a CancellationToken.

notable sequencer-relayer changes:

axum underwent heavy changes and has replaced
axum::Server by axum::serve::Serve future. This
patch wraps the future to avoid having to name its
type, erasing it, while still giving access to the
bound tcp socket address. Because its constructor
takes a tokio::net::TcpListener, the API server
setup is now async, which in turn required
SequencerRelayer::new to become async.
Due to type erasure + wrapping, the shutdown
signal closure now has to be passed in early on
instead of right before entering
sequencer-relayer's runloop. While doing so,
the oneshot channel was replaced by a cancellation
token.

Because Celestia blobs now require an app version,
the relayer write path now sets it to version 3
(because astria mainnet is targetting celestia-app
3.2.0, and the consensus node is targetting
3.3.0).

In tests, isahc was replaced by reqwest. The
entire workspace is migrating to hyper 1+ and
http 1+ and isahc has not seen upgrades in several
months and there is no work underway to make it
compatible with http 1+.

dependency tower was removed because it was not
used.

notable auctioneer changes:

there was no need for the direct dependencies
on http-body and http-body-util because tonic
already provided reexports and/or type aliases.

notable telemetry changes:

opentelemetry had a couple of breaking changes
that required reworking how the otel pipeline
is set up.

the most visible change is opentelemetry-stdout
no longer allowing to write to stderr or another
sink. because the otel stdout writer was never
used and because opentelemetry-stdout itself
notes that is should not be used in production
contexts this patch removes it entirely (and
with it the the need to explicitly set pretty
printed logs; if services are connected to a
TTY or the FORCE_STDOUT flag is set, they will
always write to stdout).
@SuperFluffy SuperFluffy force-pushed the superfluffy/tendermint-0.40 branch from bbe36c1 to b8d771b Compare February 28, 2025 17:32
@SuperFluffy SuperFluffy changed the title Superfluffy/tendermint 0.40 chore: bump all major dependencies Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cd composer pertaining to composer conductor pertaining to the astria-conductor crate override-freeze proto pertaining to the Astria Protobuf spec sequencer pertaining to the astria-sequencer crate sequencer-relayer pertaining to the astria-sequencer-relayer crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant