Skip to content

Commit

Permalink
ci/deploy: quick fix (#3649)
Browse files Browse the repository at this point in the history
1. Moving the zones onto the U.2 devices (#3557), real or synthetic,
results in the paths of all the zones changing, which results in the
paths of all their logs changing. Updated the deploy.sh job to look in
the new spot for logs, so that we can find:
2. The end-to-end test is failing[^1] because Nexus is returning a 500
on disk creation, because [Nexus cannot contact the Crucible
downstairs](https://buildomat.eng.oxide.computer/wg/0/artefact/01H5ED4P9ZPW22RMY4BEDV0X6Q/VZmMOazlZARWMoMr6qgqt59i4NHEwei5lZ4Ds8d5TJLKdbd2/01H5ED53S5T9XSX4PXS7K6GZ1S/01H5EGRG8XW9GWBQ6ZQXP93WPD/oxide-nexus:default.log?format=x-bunyan#L3759),
because [the Crucible agent is repeatedly panicking because it cannot
create a dataset, because the zpool is out of
space](https://buildomat.eng.oxide.computer/wg/0/artefact/01H5ED4P9ZPW22RMY4BEDV0X6Q/VZmMOazlZARWMoMr6qgqt59i4NHEwei5lZ4Ds8d5TJLKdbd2/01H5ED53S5T9XSX4PXS7K6GZ1S/01H5EGRF4V6N2XS8TXN2B6CK15/oxide-crucible-agent:default.log?format=x-bunyan#L93).
We attempt to rectify the issue by increasing the size of the synthetic
drives in create_virtual_hardware.sh.
3. It is possible that we are hitting this limit for the first time
because Crucible as of #3646 reserves more space.

(We should also switch the deploy job to using real disks, instead of
tmpfs, for these datasets. But that will not be part of this PR.)

[^1]: Not always; some commits are evidently lucky.
  • Loading branch information
iliana authored Jul 16, 2023
1 parent b165a21 commit 344773a
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 4 deletions.
6 changes: 3 additions & 3 deletions .github/buildomat/jobs/deploy.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
#: target = "lab-2.0-opte-0.23"
#: output_rules = [
#: "%/var/svc/log/oxide-sled-agent:default.log",
#: "%/zone/oxz_*/root/var/svc/log/oxide-*.log",
#: "%/zone/oxz_*/root/var/svc/log/system-illumos-*.log",
#: "!/zone/oxz_propolis-server_*/root/var/svc/log/*.log"
#: "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log",
#: "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log",
#: "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log"
#: ]
#: skip_clone = true
#:
Expand Down
6 changes: 5 additions & 1 deletion tools/create_virtual_hardware.sh
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,11 @@ function ensure_zpools {
echo "Zpool: [$ZPOOL]"
VDEV_PATH="$OMICRON_TOP/$ZPOOL.vdev"
if ! [[ -f "$VDEV_PATH" ]]; then
dd if=/dev/zero of="$VDEV_PATH" bs=1 count=0 seek=6G
if [[ $ZPOOL == oxp_* ]]; then
dd if=/dev/zero of="$VDEV_PATH" bs=1 count=0 seek=8G
else
dd if=/dev/zero of="$VDEV_PATH" bs=1 count=0 seek=6G
fi
fi
success "ZFS vdev $VDEV_PATH exists"
if [[ -z "$(zpool list -o name | grep $ZPOOL)" ]]; then
Expand Down

0 comments on commit 344773a

Please sign in to comment.