From 6ef40c844a8858c294862dc56a5adc18ff4cc36c Mon Sep 17 00:00:00 2001 From: Cian Hatton Date: Fri, 22 Jul 2022 14:47:50 +0100 Subject: [PATCH 1/4] chore: adding README.md for how to write e2e testse --- e2e/README.md | 315 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 315 insertions(+) create mode 100644 e2e/README.md diff --git a/e2e/README.md b/e2e/README.md new file mode 100644 index 00000000000..2f5cd3546fd --- /dev/null +++ b/e2e/README.md @@ -0,0 +1,315 @@ + +## Table of Contents +1. [How To Write Tests](#how-to-write-tests) + a. [Adding A New Test](#adding-a-new-test) + b. [Running Tests Locally](#running-tests-locally) + b. [Code Samples](#code-samples) + - [Setup](#setup) + - [Creating Test Users](#creating-test-users) + - [Waiting](#waiting) + - [Query Wallet Balances](#query-wallet-balances) + - [Broadcasting Messages](#broadcasting-messages) + - [Starting the Relayer](#starting-the-relayer) + - [Arbitrary Commands](#arbitrary-commands) + - [IBC Transfer](#ibc-transfer) +2. [Test Design](#test-design) + a. [ibctest](#ibctest) + b. [CI Configuration](#ci-configuration) +3. [Troubleshooting](#troubleshooting) + + +## How To Write Tests + +### Adding A New Test + +All tests should go under the [e2e](https://github.com/cosmos/ibc-go/tree/main/e2e) directory. When adding a new test, either add a new test function +to an existing test suite **_in the same file_**, or create a new test suite in a new file and add test functions there. + +New test suites should be composed of `testsuite.E2ETestSuite`. This type has lots of useful helper functionality that will +be quite common in most tests. + +> Note: see [here](#how-tests-are-run) for details about these requirements. + + +### Running Tests Locally + +Tests can be run using a Makefile target under the e2e directory. `e2e/Makefile` + +There are several envinronment variables that alter the behaviour of the make target. + +| Environment Variable | Description | Default Value| +| ----------- | ----------- | ----------- | +| SIMD_IMAGE | The image that will be used for simd | ibc-go-simd-e2e | +| SIMD_TAG | The tag used for simd | latest| +| RLY_TAG | The tag used for the go relayer | main| + + +> Note: when running tests locally, **no images are pushed** to the `ghcr.io/cosmos/ibc-go-simd-e2e` registry. +The images which are used only exist on your machine. + +These environment variables allow us to run tests with arbitrary verions (from branches or released) of simd +and the go relayer. + +Every time changes are pushed to a branch or to `main`, a new `simd` image is built and pushed [here](https://github.com/cosmos/ibc-go/pkgs/container/ibc-go-simd-e2e). + + +#### Example Command: + +```sh +export SIMD_IMAGE="ghcr.io/cosmos/ibc-go-simd-e2e" +export SIMD_TAG="pr-1650" +export RLY_TAG="v2.0.0-rc2" +make e2e-test suite=FeeMiddlewareTestSuite test=TestMultiMsg_MsgPayPacketFeeSingleSender +``` + + +> Note: sometimes it can be useful to make changes to [ibctest](https://github.com/strangelove-ventures/ibctest) when running tests locally. In order to do this, add the following line to +e2e/go.mod + +`replace github.com/strangelove-ventures/ibctest => ../ibctest` + +Or point it to any local checkout you have. + +### Code Samples + +#### Setup + +Every standard test will start with this. This creates two chains and a relayer, +initializes relayer accounts on both chains, establishes a connection and a channel +between the chains. + +Both chains have started, but the relayer is not yet started. + +The relayer should be started as part of the test if required. See [Starting the Relayer](#starting-the-relayer) + +```go +relayer, channelA := s.SetupChainsRelayerAndChannel(ctx, feeMiddlewareChannelOptions()) +chainA, chainB := s.GetChains() +``` + +#### Creating Test Users + +There are helper functions to easily create users on both chains. + +```go +chainAWallet := s.CreateUserOnChainA(ctx, testvalues.StartingTokenAmount) +chainBWallet := s.CreateUserOnChainB(ctx, testvalues.StartingTokenAmount) +``` + +#### Waiting + +We can wait for some number of blocks on the specified chains if required. + +```go +chainA, chainB := s.GetChains() +err := test.WaitForBlocks(ctx, 1, chainA, chainB) +s.Require().NoError(err) +``` + +#### Query Wallet Balances + +We can fetch balances of wallets on specific chains. + +```go +chainABalance, err := s.GetChainANativeBalance(ctx, chainAWallet) +s.Require().NoError(err) +``` + +#### Broadcasting Messages + +We can broadcast arbitrary messages which are signed on behalf of users created in the test. + +This example shows a multi message transaction being broadcast on chainA and signed on behalf of chainAWallet. + +```go +relayer, channelA := s.SetupChainsRelayerAndChannel(ctx, feeMiddlewareChannelOptions()) +chainA, chainB := s.GetChains() + +chainAWallet := s.CreateUserOnChainA(ctx, testvalues.StartingTokenAmount) +chainBWallet := s.CreateUserOnChainB(ctx, testvalues.StartingTokenAmount) + +t.Run("broadcast multi message transaction", func(t *testing.T){ + payPacketFeeMsg := feetypes.NewMsgPayPacketFee(testFee, channelA.PortID, channelA.ChannelID, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), nil) + transferMsg := transfertypes.NewMsgTransfer(channelA.PortID, channelA.ChannelID, transferAmount, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), chainBWallet.Bech32Address(chainB.Config().Bech32Prefix), clienttypes.NewHeight(1, 1000), 0) + resp, err := s.BroadcastMessages(ctx, chainA, chainAWallet, payPacketFeeMsg, transferMsg) + + s.AssertValidTxResponse(resp) + s.Require().NoError(err) +}) +``` + +#### Starting the Relayer + +The relayer can be started with the following. + +```go +t.Run("start relayer", func(t *testing.T) { + s.StartRelayer(relayer) +}) +``` + +#### Arbitrary Commands + +Arbitrary commands can be executed on a given chain. + +> Note: these commands will be fully configured to run on the chain executed on (home directory, ports configured etc.) + +However, it is preferable to [broadcast messages](#broadcasting-messages) or use a gRPC query if possible. + +```go +stdout, stderr, err := chainA.Exec(ctx, []string{"tx", "..."}, nil) +``` + +#### IBC Transfer + +It is possible to send an IBC transfer in two ways. + +Use the ibctest Chain interface (this ultimately does a docker exec) + +```go +t.Run("send IBC transfer", func(t *testing.T) { + chainATx, err = chainA.SendIBCTransfer(ctx, channelA.ChannelID, chainAWallet.KeyName, walletAmount, nil) + s.Require().NoError(err) + s.Require().NoError(chainATx.Validate(), "chain-a ibc transfer tx is invalid") +}) +``` + +Broadcast a `MsgTransfer`. + +```go +t.Run("send IBC transfer", func(t *testing.T){ + transferMsg := transfertypes.NewMsgTransfer(channelA.PortID, channelA.ChannelID, transferAmount, chainAWallet.Bech32Address(chainA.Config().Bech32Prefix), chainBWallet.Bech32Address(chainB.Config().Bech32Prefix), clienttypes.NewHeight(1, 1000), 0) + resp, err := s.BroadcastMessages(ctx, chainA, chainAWallet, transferMsg) + s.AssertValidTxResponse(resp) + s.Require().NoError(err) +}) +``` + +## Test Design + + +#### ibctest + +These E2E tests use the [ibctest framework](https://github.com/strangelove-ventures/ibctest). This framework creates chains and relayers in containers and allows for arbitrary commands to be executed in the chain containers, +as well as allowing us to broadcast arbitrary messages which are signed on behalf of a user created in the test. + + +#### CI Configuration + +There are two main github actions for e2e tests. + +[e2e.yaml](https://github.com/cosmos/ibc-go/blob/main/.github/workflows/e2e.yaml) which runs when collaborators create branches. + +[e2e-fork.yaml](https://github.com/cosmos/ibc-go/blob/main/.github/workflows/e2e-fork.yml) which runs when forks are created. + +In `e2e.yaml`, the `simd` image is built and pushed to [a registry](https://github.com/cosmos/ibc-go/pkgs/container/ibc-go-simd-e2e) and every test +that is run uses the image that was built. + +In `e2e-fork.yaml`, images are not pushed to this registry, but instead remain local to the host runner. + + +### How tests are run + +The tests use the [matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) feature of Github Actions. The matrix is +dynamically generated using [this script](https://github.com/cosmos/ibc-go/blob/main/.github/scripts/build_test_matrix.go). + +> Note: there is currently a limitation that all tests belonging to a test suite must be in the same file. +In order to support test functions spread in different files, we would either need to manually maintain a matrix +or update the script to account for this. The script assumes there is a single test suite per test file to avoid an overly complex +generation process. + +Which looks under the `e2e` directory, and creates a task for each test suite function. + +#### Example + +```go +// e2e/file_one_test.go +package e2e + +func TestFeeMiddlewareTestSuite(t *testing.T) { + suite.Run(t, new(FeeMiddlewareTestSuite)) +} + +type FeeMiddlewareTestSuite struct { + testsuite.E2ETestSuite +} + +func (s *FeeMiddlewareTestSuite) TestA() {} +func (s *FeeMiddlewareTestSuite) TestB() {} +func (s *FeeMiddlewareTestSuite) TestC() {} + +``` + +```go +// e2e/file_two_test.go +package e2e + +func TestTransferTestSuite(t *testing.T) { + suite.Run(t, new(TransferTestSuite)) +} + +type TransferTestSuite struct { + testsuite.E2ETestSuite +} + +func (s *TransferTestSuite) TestD() {} +func (s *TransferTestSuite) TestE() {} +func (s *TransferTestSuite) TestF() {} + +``` + +In the above example, the following would be generated. + +```json +{ + "include": [ + { + "suite": "FeeMiddlewareTestSuite", + "test": "TestA" + }, + { + "suite": "FeeMiddlewareTestSuite", + "test": "TestB" + }, + { + "suite": "FeeMiddlewareTestSuite", + "test": "TestC" + }, + { + "suite": "TransferTestSuite", + "test": "TestD" + }, + { + "suite": "TransferTestSuite", + "test": "TestE" + }, + { + "suite": "TransferTestSuite", + "test": "TestF" + } + ] +} +``` + +This string is used to generate a test matrix in the Github Action that runs the E2E tests. + +All tests will be run on different hosts. + + +#### Misceleneous: + +* Gas fees are set to zero to simply calcuations when asserting account balances. +* When upgrading from e.g. v4 -> v5, in ibc-go, we cannot upgrade the go.mod under `e2e` since v5 will not yet exist. We need to upgrade it in a follow up PR. + +### Troubleshooting + +* On Mac, after running a lot of tests, it can happen that containers start failing. To fix this, you can try clearing existing containers and restarting the docker daemon. + + This generally manifests itself as relayer or simd containers timing out during setup stages of the test. This doesn't happen in CI. + ```bash + # delete all images + docker system prune -af + ``` + This issue doesn't seem to occur on other operating systems. + + From 90ac4b7a8a073040d1b797db05a840461274591b Mon Sep 17 00:00:00 2001 From: Cian Hatton Date: Fri, 22 Jul 2022 15:48:58 +0100 Subject: [PATCH 2/4] removing link to github docs --- e2e/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/e2e/README.md b/e2e/README.md index 2f5cd3546fd..8f07de1bdc2 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -210,7 +210,7 @@ In `e2e-fork.yaml`, images are not pushed to this registry, but instead remain l ### How tests are run -The tests use the [matrix](https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs) feature of Github Actions. The matrix is +The tests use the `matrix` feature of Github Actions. The matrix is dynamically generated using [this script](https://github.com/cosmos/ibc-go/blob/main/.github/scripts/build_test_matrix.go). > Note: there is currently a limitation that all tests belonging to a test suite must be in the same file. From d6260fff0da6f7b85c6b1e7105f580474955d230 Mon Sep 17 00:00:00 2001 From: Cian Hatton Date: Thu, 28 Jul 2022 10:13:26 +0100 Subject: [PATCH 3/4] chore: addressing PR feedback --- e2e/README.md | 31 ++++++++++++++++--------------- 1 file changed, 16 insertions(+), 15 deletions(-) diff --git a/e2e/README.md b/e2e/README.md index 8f07de1bdc2..00448cd84dc 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -1,20 +1,20 @@ ## Table of Contents 1. [How To Write Tests](#how-to-write-tests) - a. [Adding A New Test](#adding-a-new-test) - b. [Running Tests Locally](#running-tests-locally) - b. [Code Samples](#code-samples) - - [Setup](#setup) - - [Creating Test Users](#creating-test-users) - - [Waiting](#waiting) - - [Query Wallet Balances](#query-wallet-balances) - - [Broadcasting Messages](#broadcasting-messages) - - [Starting the Relayer](#starting-the-relayer) - - [Arbitrary Commands](#arbitrary-commands) - - [IBC Transfer](#ibc-transfer) + - a. [Adding A New Test](#adding-a-new-test) + - b. [Running Tests Locally](#running-tests-locally) + - b. [Code Samples](#code-samples) + - [Setup](#setup) + - [Creating Test Users](#creating-test-users) + - [Waiting](#waiting) + - [Query Wallet Balances](#query-wallet-balances) + - [Broadcasting Messages](#broadcasting-messages) + - [Starting the Relayer](#starting-the-relayer) + - [Arbitrary Commands](#arbitrary-commands) + - [IBC Transfer](#ibc-transfer) 2. [Test Design](#test-design) - a. [ibctest](#ibctest) - b. [CI Configuration](#ci-configuration) + - a. [ibctest](#ibctest) + - b. [CI Configuration](#ci-configuration) 3. [Troubleshooting](#troubleshooting) @@ -24,6 +24,7 @@ All tests should go under the [e2e](https://github.com/cosmos/ibc-go/tree/main/e2e) directory. When adding a new test, either add a new test function to an existing test suite **_in the same file_**, or create a new test suite in a new file and add test functions there. +New test files should follow the convention of `module_name_test.go`. New test suites should be composed of `testsuite.E2ETestSuite`. This type has lots of useful helper functionality that will be quite common in most tests. @@ -164,7 +165,7 @@ stdout, stderr, err := chainA.Exec(ctx, []string{"tx", "..."}, nil) It is possible to send an IBC transfer in two ways. -Use the ibctest Chain interface (this ultimately does a docker exec) +Use the ibctest `Chain` interface (this ultimately does a docker exec) ```go t.Run("send IBC transfer", func(t *testing.T) { @@ -211,7 +212,7 @@ In `e2e-fork.yaml`, images are not pushed to this registry, but instead remain l ### How tests are run The tests use the `matrix` feature of Github Actions. The matrix is -dynamically generated using [this script](https://github.com/cosmos/ibc-go/blob/main/.github/scripts/build_test_matrix.go). +dynamically generated using [this command](https://github.com/cosmos/ibc-go/blob/main/cmd/build_test_matrix/main.go). > Note: there is currently a limitation that all tests belonging to a test suite must be in the same file. In order to support test functions spread in different files, we would either need to manually maintain a matrix From c7f32e06cbcd4e4b2d5db11d63a99e15345a9896 Mon Sep 17 00:00:00 2001 From: Cian Hatton Date: Thu, 28 Jul 2022 10:17:39 +0100 Subject: [PATCH 4/4] chore: changing titles to use Google style guide --- e2e/README.md | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/e2e/README.md b/e2e/README.md index 00448cd84dc..4475891db22 100644 --- a/e2e/README.md +++ b/e2e/README.md @@ -1,26 +1,26 @@ ## Table of Contents -1. [How To Write Tests](#how-to-write-tests) - - a. [Adding A New Test](#adding-a-new-test) - - b. [Running Tests Locally](#running-tests-locally) - - b. [Code Samples](#code-samples) +1. [How to write tests](#how-to-write-tests) + - a. [Adding a new test](#adding-a-new-test) + - b. [Running tests locally](#running-tests-locally) + - b. [Code samples](#code-samples) - [Setup](#setup) - - [Creating Test Users](#creating-test-users) + - [Creating test users](#creating-test-users) - [Waiting](#waiting) - - [Query Wallet Balances](#query-wallet-balances) - - [Broadcasting Messages](#broadcasting-messages) - - [Starting the Relayer](#starting-the-relayer) - - [Arbitrary Commands](#arbitrary-commands) - - [IBC Transfer](#ibc-transfer) -2. [Test Design](#test-design) + - [Query wallet balances](#query-wallet-balances) + - [Broadcasting messages](#broadcasting-messages) + - [Starting the relayer](#starting-the-relayer) + - [Arbitrary commands](#arbitrary-commands) + - [IBC transfer](#ibc-transfer) +2. [Test design](#test-design) - a. [ibctest](#ibctest) - - b. [CI Configuration](#ci-configuration) + - b. [CI configuration](#ci-configuration) 3. [Troubleshooting](#troubleshooting) -## How To Write Tests +## How to write tests -### Adding A New Test +### Adding a new test All tests should go under the [e2e](https://github.com/cosmos/ibc-go/tree/main/e2e) directory. When adding a new test, either add a new test function to an existing test suite **_in the same file_**, or create a new test suite in a new file and add test functions there. @@ -32,7 +32,7 @@ be quite common in most tests. > Note: see [here](#how-tests-are-run) for details about these requirements. -### Running Tests Locally +### Running tests locally Tests can be run using a Makefile target under the e2e directory. `e2e/Makefile` @@ -71,7 +71,7 @@ e2e/go.mod Or point it to any local checkout you have. -### Code Samples +### Code samples #### Setup @@ -88,7 +88,7 @@ relayer, channelA := s.SetupChainsRelayerAndChannel(ctx, feeMiddlewareChannelOpt chainA, chainB := s.GetChains() ``` -#### Creating Test Users +#### Creating test users There are helper functions to easily create users on both chains. @@ -107,7 +107,7 @@ err := test.WaitForBlocks(ctx, 1, chainA, chainB) s.Require().NoError(err) ``` -#### Query Wallet Balances +#### Query wallet balances We can fetch balances of wallets on specific chains. @@ -116,7 +116,7 @@ chainABalance, err := s.GetChainANativeBalance(ctx, chainAWallet) s.Require().NoError(err) ``` -#### Broadcasting Messages +#### Broadcasting messages We can broadcast arbitrary messages which are signed on behalf of users created in the test. @@ -139,7 +139,7 @@ t.Run("broadcast multi message transaction", func(t *testing.T){ }) ``` -#### Starting the Relayer +#### Starting the relayer The relayer can be started with the following. @@ -149,7 +149,7 @@ t.Run("start relayer", func(t *testing.T) { }) ``` -#### Arbitrary Commands +#### Arbitrary commands Arbitrary commands can be executed on a given chain. @@ -161,7 +161,7 @@ However, it is preferable to [broadcast messages](#broadcasting-messages) or use stdout, stderr, err := chainA.Exec(ctx, []string{"tx", "..."}, nil) ``` -#### IBC Transfer +#### IBC transfer It is possible to send an IBC transfer in two ways. @@ -186,7 +186,7 @@ t.Run("send IBC transfer", func(t *testing.T){ }) ``` -## Test Design +## Test design #### ibctest @@ -195,7 +195,7 @@ These E2E tests use the [ibctest framework](https://github.com/strangelove-ventu as well as allowing us to broadcast arbitrary messages which are signed on behalf of a user created in the test. -#### CI Configuration +#### CI configuration There are two main github actions for e2e tests.