omicron-dev -- db-run not working on Helios #475

smklein · 2021-12-02T18:46:44Z

Since #427 landed, if I attempt to run:

$ cargo run --bin=omicron-dev -- db-run

I see the following:

   Compiling omicron-test-utils v0.1.0 (/home/smklein/omicron/test-utils)
    Finished dev [unoptimized + debuginfo] target(s) in 5.49s
     Running `target/debug/omicron-dev db-run`
omicron-dev: using temporary directory for database store (cleaned up on clean exit)
omicron-dev: will run this to start CockroachDB:
cockroach start-single-node --insecure --http-addr=:0 --store /tmp/.tmpPAeMXd/data --listen-addr 127.0.0.1:32221 --listening-url-file /tmp/.tmpPAeMXd/listen-url
omicron-dev: temporary directory: /tmp/.tmpPAeMXd
omicron-dev: wrong version of CockroachDB installed. expected 'v21.1.10', found: 'Err(error Some(4) when checking CockroachDB version)

This is true even after re-running the ./tools/ci_download_cockroachdb script.

The text was updated successfully, but these errors were encountered:

smklein · 2021-12-02T18:46:56Z

FYI @teisenbe

teisenbe · 2021-12-02T18:50:09Z

I can go take a look, sorry about that

teisenbe · 2021-12-02T18:50:28Z

(I'll revert the version check for now)

smklein · 2021-12-02T18:53:18Z

No worries, I'll see if I can scrape together some extra info for debugging:

Running ./tools/ci_download_cockroachdb eventually emits the following:

Build Tag:        v20.2.5
Build Time:       2021/03/17 21:00:51
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.2
C Compiler:       gcc 9.3.0
Build Commit ID:  162c5ac4968cf31c0ed54cd29aa8aeccd66247bb
Build Type:       release

Notably, the build tag is not 21.1.10 - it's 20.2.5.

Also, running: $ ./cockroachdb/bin/cockroach version --build-tag shows the following:

Usage:
  cockroach version [flags]

Flags:
  -h, --help   help for version

Global Flags:
      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
      --no-color                         disable standard error log colorization
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)

ERROR: unknown flag: --build-tag
Failed running "version"

... so, although the check might be sorta broken right now, there is actually a mismatch in CRDB versions here!

teisenbe · 2021-12-02T18:54:18Z

oh, huh, I wonder if build tag is a new argument in CRDB

teisenbe · 2021-12-02T18:56:40Z

Oh, huh. Off by just one minor rev:
https://www.cockroachlabs.com/docs/releases/v20.2.6#command-line-changes. So it looks like the logic likely works, and Helios is just so far stale that the version check doesn't execute as expected

teisenbe · 2021-12-02T19:06:05Z

I can't reproduce over here. Running tools/ci_download_cockroachdb from main on my Helios VM gets me

Build Tag:        v21.1.10
Build Time:       2021/10/12 22:52:33
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.5
C Compiler:       gcc 10.3.0
Build Commit ID:  a6daab16ee8a1c15abc4c4a8a425e46d12033b5c
Build Type:       release

I'm not sure how it'd end up trying to fetch such an old version given how recent the version check logic is. Any chance you have local changes to tools/ci_download_cockroachdb? The error message from db-run suggests that tools/cockroachdb_version is present and correct

davepacheco · 2021-12-02T19:06:11Z

@smklein can you paste the full output of ci_download_cockroachdb? When I run it from commit 41c10d6 (tip of main) I get:

$ ./tools/ci_download_cockroachdb 
+ set -o errexit
+++ dirname ./tools/ci_download_cockroachdb
++ cd ./tools
++ pwd
+ SOURCE_DIR=/home/dap/omicron/tools
++ basename ./tools/ci_download_cockroachdb
+ ARG0=ci_download_cockroachdb
++ cat /home/dap/omicron/tools/cockroachdb_version
+ CIDL_VERSION=v21.1.10
+ source /home/dap/omicron/tools/cockroachdb_checksums
++ CIDL_MD5_DARWIN=cab1998101cc7138c52c312aff0776dc
++ CIDL_MD5_LINUX=d0aab99810ddf37b2f6d183077df262d
++ CIDL_MD5_ILLUMOS=499b7f9a80028c993b87808556c29425
+ CIDL_ASSEMBLE_DIR=./cockroachdb
+ CIDL_URL_COCKROACH=https://binaries.cockroachdb.com
+ CIDL_URL_ILLUMOS=https://illumos.org/downloads
+ main
+ [[ 0 != 0 ]]
+ CIDL_OS=solaris2.11
+ [[ 0 != 0 ]]
+ configure_os solaris2.11
+ echo 'current directory: /home/dap/omicron'
current directory: /home/dap/omicron
+ echo 'configuring based on OS: "solaris2.11"'
configuring based on OS: "solaris2.11"
+ case "$1" in
+ CIDL_BUILD=illumos
+ CIDL_SUFFIX=tar.gz
+ CIDL_MD5=499b7f9a80028c993b87808556c29425
+ CIDL_MD5FUNC=do_md5sum
+ CIDL_URL_BASE=https://illumos.org/downloads
+ CIDL_ASSEMBLE=do_assemble_illumos
+ CIDL_DIR=cockroach-v21.1.10.illumos
+ CIDL_FILE=cockroach-v21.1.10.illumos.tar.gz
+ CIDL_URL=https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz'
URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'Local file: cockroach-v21.1.10.illumos.tar.gz'
Local file: cockroach-v21.1.10.illumos.tar.gz
+ do_download_curl https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz cockroach-v21.1.10.illumos.tar.gz
+ curl --silent --show-error --fail --location --output cockroach-v21.1.10.illumos.tar.gz https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
++ do_md5sum cockroach-v21.1.10.illumos.tar.gz
++ md5sum
++ awk '{print $1}'
+ calculated_md5=499b7f9a80028c993b87808556c29425
+ [[ 499b7f9a80028c993b87808556c29425 != \4\9\9\b\7\f\9\a\8\0\0\2\8\c\9\9\3\b\8\7\8\0\8\5\5\6\c\2\9\4\2\5 ]]
+ do_untar cockroach-v21.1.10.illumos.tar.gz
+ tar xzf cockroach-v21.1.10.illumos.tar.gz
+ do_assemble_illumos cockroach-v21.1.10.illumos
+ cp -r cockroach-v21.1.10 ./cockroachdb
+ ./cockroachdb/bin/cockroach version
Build Tag:        v21.1.10
Build Time:       2021/10/12 22:52:33
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.5
C Compiler:       gcc 10.3.0
Build Commit ID:  a6daab16ee8a1c15abc4c4a8a425e46d12033b5c
Build Type:       release

That looks right to me but it's different than you saw.

smklein · 2021-12-02T19:08:43Z

$ ./tools/ci_download_cockroachdb 
+ set -o errexit
+++ dirname ./tools/ci_download_cockroachdb
++ cd ./tools
++ pwd
+ SOURCE_DIR=/home/smklein/omicron/tools
++ basename ./tools/ci_download_cockroachdb
+ ARG0=ci_download_cockroachdb
++ cat /home/smklein/omicron/tools/cockroachdb_version
+ CIDL_VERSION=v21.1.10
+ source /home/smklein/omicron/tools/cockroachdb_checksums
++ CIDL_MD5_DARWIN=cab1998101cc7138c52c312aff0776dc
++ CIDL_MD5_LINUX=d0aab99810ddf37b2f6d183077df262d
++ CIDL_MD5_ILLUMOS=499b7f9a80028c993b87808556c29425
+ CIDL_ASSEMBLE_DIR=./cockroachdb
+ CIDL_URL_COCKROACH=https://binaries.cockroachdb.com
+ CIDL_URL_ILLUMOS=https://illumos.org/downloads
+ main
+ [[ 0 != 0 ]]
+ CIDL_OS=solaris2.11
+ [[ 0 != 0 ]]
+ configure_os solaris2.11
+ echo 'current directory: /home/smklein/omicron'
current directory: /home/smklein/omicron
+ echo 'configuring based on OS: "solaris2.11"'
configuring based on OS: "solaris2.11"
+ case "$1" in
+ CIDL_BUILD=illumos
+ CIDL_SUFFIX=tar.gz
+ CIDL_MD5=499b7f9a80028c993b87808556c29425
+ CIDL_MD5FUNC=do_md5sum
+ CIDL_URL_BASE=https://illumos.org/downloads
+ CIDL_ASSEMBLE=do_assemble_illumos
+ CIDL_DIR=cockroach-v21.1.10.illumos
+ CIDL_FILE=cockroach-v21.1.10.illumos.tar.gz
+ CIDL_URL=https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz'
URL: https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
+ echo 'Local file: cockroach-v21.1.10.illumos.tar.gz'
Local file: cockroach-v21.1.10.illumos.tar.gz
+ do_download_curl https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz cockroach-v21.1.10.illumos.tar.gz
+ curl --silent --show-error --fail --location --output cockroach-v21.1.10.illumos.tar.gz https://illumos.org/downloads/cockroach-v21.1.10.illumos.tar.gz
++ do_md5sum cockroach-v21.1.10.illumos.tar.gz
++ md5sum
++ awk '{print $1}'
+ calculated_md5=499b7f9a80028c993b87808556c29425
+ [[ 499b7f9a80028c993b87808556c29425 != \4\9\9\b\7\f\9\a\8\0\0\2\8\c\9\9\3\b\8\7\8\0\8\5\5\6\c\2\9\4\2\5 ]]
+ do_untar cockroach-v21.1.10.illumos.tar.gz
+ tar xzf cockroach-v21.1.10.illumos.tar.gz
+ do_assemble_illumos cockroach-v21.1.10.illumos
+ cp -r cockroach-v21.1.10 ./cockroachdb
+ ./cockroachdb/bin/cockroach version
Build Tag:        v20.2.5
Build Time:       2021/03/17 21:00:51
Distribution:     OSS
Platform:         illumos amd64 (x86_64-pc-solaris2.11)
Go Version:       go1.16.2
C Compiler:       gcc 9.3.0
Build Commit ID:  162c5ac4968cf31c0ed54cd29aa8aeccd66247bb
Build Type:       release

teisenbe · 2021-12-02T19:10:54Z

Oh interesting. It tried downloading 21.1.10, but got 20.2.5? I wonder if the last bits of the script behave incorrectly on Helios

teisenbe · 2021-12-02T19:12:00Z

I'm gonna check out an old rev to get 20.2.5 and then go back to main and try to repro

Edit: Didn't even have to get that far. Just checking out an old rev is enough. The same issue happens, just that it fails to replace 21.1.10 with the older 20.2.5

smklein · 2021-12-02T19:12:15Z

Ugh, okay, I just rm -rf'd cockroachdb and re-ran, and I'm seeing the 21.1.10 version now. I'm still not sure why this is the case.

leftwo · 2021-12-02T19:14:30Z

I saw this last night on broadway.
If the ./cockroach directory exists when you run ./tools/ci_download_cockroachdb it will put the new version inside instead of replacing the existing version.

teisenbe · 2021-12-02T19:15:13Z

Just saw the same behavior leftwo describes. I'll send a patch for the script

teisenbe mentioned this issue Dec 2, 2021

Revert "dev: Make db-run check CockroachDB version (#427)" #476

Closed

teisenbe mentioned this issue Dec 2, 2021

cockroachdb: Fix download script for illumos #477

Merged

teisenbe closed this as completed in #477 Dec 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

omicron-dev -- db-run not working on Helios #475

omicron-dev -- db-run not working on Helios #475

smklein commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

davepacheco commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021 •

edited

Loading

smklein commented Dec 2, 2021 •

edited

Loading

leftwo commented Dec 2, 2021

teisenbe commented Dec 2, 2021

omicron-dev -- db-run not working on Helios #475

omicron-dev -- db-run not working on Helios #475

Comments

smklein commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021

davepacheco commented Dec 2, 2021

smklein commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021 • edited Loading

smklein commented Dec 2, 2021 • edited Loading

leftwo commented Dec 2, 2021

teisenbe commented Dec 2, 2021

teisenbe commented Dec 2, 2021 •

edited

Loading

smklein commented Dec 2, 2021 •

edited

Loading