testing / Gateway e2e test #75

ghost · 2023-09-20T11:11:11Z

includes additional changes:

update command babylonnode docker stop to not depend on the config.yaml but only on the specified docker-compose.yaml. Avoiding situations where you could not stop the docker-compose using the cli when the config.yaml has been changed.
removal of unused question DockerSetup.get_existing_compose_file(docker_config) in DETAILED mode. File was asked and loaded but no information was stored or transformed.
removal of forced platform: linux/amd64 in docker-compose template
Local unit test for PROMPT_FEEDER for quicker feedback than CI tests.
Extracting e2e tests for systemd and gateway setup into bash scripts for manual execution on fresh nodes
Adding of PROMPT_FEED for migration_release
Fix: respecting user input over defaults (was not the case when asking keystore directory or data directory)
Fix: actually error out when scripts fail. run_shell_command was previously just printing to exit but never actually exited.
Use common dependency function for docker dependencies and systemd dependencies to avoid installation of different docker runtime environments or forgetting ansible installation

github-actions · 2023-09-20T11:12:43Z

Test Results

45 tests +1 39 ✔️ +1 18s ⏱️ -5s
  1 suites ±0   6 💤 ±0
  1 files ±0   0 ❌ ±0

Results for commit 00d2679. ± Comparison against base commit 913fed9.

♻️ This comment has been updated with latest results.

github-actions · 2023-09-20T11:12:45Z

☂️ Python Coverage

current status: ✅

Overall Coverage

Lines	Covered	Coverage	Threshold	Status
3637	2361	65%	0%	🟢

New Files

File	Coverage	Status
node-runner-cli/tests/unit/test_prompt_feeder_local.py	95%	🟢
TOTAL	95%	🟢

Modified Files

File	Coverage	Status
node-runner-cli/commands/dockercommand.py	48%	🟢
node-runner-cli/commands/monitoring.py	29%	🟢
node-runner-cli/commands/othercommands.py	86%	🟢
node-runner-cli/commands/systemdcommand.py	51%	🟢
node-runner-cli/setup/BaseSetup.py	53%	🟢
node-runner-cli/setup/DockerSetup.py	49%	🟢
node-runner-cli/setup/GatewaySetup.py	95%	🟢
node-runner-cli/tests/unit/test_docker.py	95%	🟢
node-runner-cli/utils/PromptFeeder.py	97%	🟢
node-runner-cli/utils/Prompts.py	67%	🟢
node-runner-cli/utils/utils.py	55%	🟢
TOTAL	66%	🟢

updated for commit: 00d2679 by action🐍

* fix docker installation when systemd * running systemd dependencies command as part of the tests * avois rootless --------- Co-authored-by: Santiago Baldassin <santiago.baldassin@rdx.works>

shambupujar · 2023-09-22T13:01:22Z

node-runner-cli/tests/userflows/install-docker-all-same-host.sh

+docker system df
+PGPASSWORD=$POSTGRES_PASSWORD psql -h localhost -U postgres -d radixdlt_ledger -P pager=off -c "\l+"
+PGPASSWORD=$POSTGRES_PASSWORD psql -h localhost -U postgres -d radixdlt_ledger -P pager=off -c "select pg_size_pretty(pg_database_size('radixdlt_ledger'));"
+PGPASSWORD=$POSTGRES_PASSWORD psql -h localhost -U postgres -d radixdlt_ledger -P pager=off -c "select pg_size_pretty(pg_database_size('radixdlt_ledger'));"


Why we running same query twice?

good point, that was supposed to be a different query

shambupujar · 2023-09-22T13:03:14Z

node-runner-cli/tests/userflows/install-docker-all-same-host.sh

+echo "Testing gateway endpoints available"
+
+echo "Gateway endpoint"
+curl -k -u "gateway:${NGINX_GATEWAY_PASSWORD}" https://localhost/gateway | jq


Does Jq fail with exit status that makes job fail?

Why running this curl commands instead of cli commands?

Those commands do not have a cli version yet since they target the gateway. Good ticket for the future actually. Adding the gateway api commands to cli.

And yes, the jq will fail if the response is not a valid json. Which is the case if the endpoint is not available, authentication failed or similar.

It wont fail if the gateway returns a valid json that states something like this {status:"not ok"}.
I dont know how that response looks like yet.

shambupujar · 2023-09-22T13:05:35Z

node-runner-cli/test-prompts/gateway-local-core-remote-postgress.yml

@@ -0,0 +1,25 @@
+---


How are we running remote postgres?

currently not. I tried with the docker container. But with no success. I did not want to delete the prompts nor the bash script yet though, because they are still useful for testing manually or using external services.

shambupujar · 2023-09-22T13:17:35Z

node-runner-cli/templates/radix-fullnode-compose.yml.j2

@@ -109,7 +108,6 @@ services:
    depends_on:
      - database_migrations
    image: {{gateway.data_aggregator.repo}}:{{gateway.data_aggregator.release}}
-    platform: linux/amd64


Not sure why we had this ? Now we have arm images, hence we removing this line?

Yes exactly. I think I added it back in February because I was running the resulting docker-compose locally and there was no arm images available, resulting in "cant find the image"-error.

Kim Fehrs added 2 commits September 20, 2023 12:06

simplify docker stop command

7442f36

add userflow test

8119087

apt install -y

c9cf7cd

github-actions bot added the ansible label Sep 20, 2023

Kim Fehrs added 20 commits September 20, 2023 13:24

experiment with hosted runners for e2e test

e47cc46

fix file path

4ad6256

try fix experiment by specifying location of docker-compose

d68c5f4

add whoami

1916f04

workflow without waiting for build

1d75131

force new workflow

47bf73d

increase wait time for core node

7cca520

add authentication to system health command

502481b

set auth depending on user thats running

f6d2026

add repository varibales to workflow

14a18fd

test fix core-node-only

4f25bf4

test fix experiment

3306621

fix variable names

17e1910

install dependent apps

6fd49af

install dependent apps

4416f37

add exit to run_shell_command

0adb8fd

install ansible

8b852e8

apt-get install -y

2be0c91

add docker logs for easier debugging

20eb404

remove command that install non existing file

8fa24e4

github-actions bot removed the ansible label Sep 20, 2023

Kim Fehrs and others added 3 commits September 20, 2023 17:44

change test to indicate docker ps health

422bd2d

Chore/fix docker install (#77)

aca3a41

* fix docker installation when systemd * running systemd dependencies command as part of the tests * avois rootless --------- Co-authored-by: Santiago Baldassin <santiago.baldassin@rdx.works>

clean up dependencies command

87b5d7c

Kim Fehrs added 21 commits September 21, 2023 17:46

install wget first

da66cb3

install wget first

3d9c928

remove sudo

b5f1214

install docker differently

af6af17

put everything in place

7fd84e7

remove redundant first time setup question

756803b

remove redundant tests

4ec6d22

skip dependency installation in container on hosted runner

6b4ae0e

do not disable postgres to avoid future tests to fail

357d373

fix workflow syntax?

d895480

remove warning about node12

75b089f

add radixdlt instead of current user to docker group

023643d

Update some prompts

00cd28e

readd old gateway config tests

878d234

disable test-userflow-docker-gateway-seperate-postgres

3712957

fix default value when prompting core api for gateway

ad145ac

do not stop postgres service

12bd212

update prompt feed for gateway tests

2c83401

add non interactive optimise_node for tests

2f4026c

fix dumb typo

cb68c6c

test optimise-node also

1d01ecc

ghost marked this pull request as ready for review September 22, 2023 12:10

remove nginx version override

4be71d9

shambupujar reviewed Sep 22, 2023

View reviewed changes

remove duplicated psql database size query

00d2679

shambupujar approved these changes Sep 22, 2023

View reviewed changes

ghost merged commit 73b45f0 into main Sep 22, 2023

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

testing / Gateway e2e test #75

testing / Gateway e2e test #75

ghost commented Sep 20, 2023 •

edited by ghost

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

shambupujar Sep 22, 2023

ghost Sep 22, 2023

shambupujar Sep 22, 2023

shambupujar Sep 22, 2023

ghost Sep 22, 2023

ghost Sep 22, 2023

shambupujar Sep 22, 2023

ghost Sep 22, 2023

shambupujar Sep 22, 2023

ghost Sep 22, 2023

testing / Gateway e2e test #75

testing / Gateway e2e test #75

Conversation

ghost commented Sep 20, 2023 • edited by ghost Loading

github-actions bot commented Sep 20, 2023 • edited Loading

Test Results

github-actions bot commented Sep 20, 2023 • edited Loading

☂️ Python Coverage

Overall Coverage

New Files

Modified Files

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Sep 20, 2023 •

edited by ghost

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading