Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

question: Would it be valuable to have "crane registry server" pass the OCI distribution conformance tests #1661

Closed
ktarplee opened this issue Apr 19, 2023 · 3 comments
Closed

question: Would it be valuable to have "crane registry server" pass the OCI distribution conformance tests #1661

ktarplee opened this issue Apr 19, 2023 · 3 comments
Labels
lifecycle/stale question Further information is requested

Comments

@ktarplee
Copy link
Contributor

The stated goals of the in-memory crane test registry are

Its goal is to be standards compliant and its strictness will increase over time.

I do find it useful to have a very strict (and conformant) registry to test against in unit tests.

To run the OCI Distribution Conformance Tests all you have to do is:

git clone https://github.com/opencontainers/distribution-spec.git
cd conformance

crane registry serve &
# extract the registry port from the output of the above command
export REGISTRY_PORT=12345

# Registry details
export OCI_ROOT_URL="http://localhost:$REGISTRY_PORT"
export OCI_NAMESPACE="myorg/myrepo"
export OCI_CROSSMOUNT_NAMESPACE="myorg/other"
export OCI_USERNAME="myuser"
export OCI_PASSWORD="mypass"

# Which workflows to run
export OCI_TEST_PULL=1
export OCI_TEST_PUSH=1
export OCI_TEST_CONTENT_DISCOVERY=1
export OCI_TEST_CONTENT_MANAGEMENT=1

# Extra settings
export OCI_HIDE_SKIPPED_WORKFLOWS=0
export OCI_DEBUG=0
export OCI_DELETE_MANIFEST_BEFORE_BLOBS=0

go test -c
conformance.test

Currently some of the tests pass but most of them fail.

$ ./conformance.test 
Running Suite: OCI Distribution Conformance Tests
=================================================
Random Seed: 1681910215
Will run 68 of 68 specs

••••
------------------------------
S [SKIPPING] [0.000 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Pull
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:13
    Setup
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:17
      Get tag name from environment [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:85

      you have skipped this test.

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/setup.go:351
------------------------------
••••••••••
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Pull
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:13
    Error codes
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:196
      400 response body should contain OCI-conforming JSON message [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:197

      Expected
          <int>: 201
      To satisfy at least one of these matchers: [%!s(*matchers.EqualMatcher=&{400}) %!s(*matchers.EqualMatcher=&{404})]

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:205
------------------------------
•••
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Streamed
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:23
      PATCH request with blob in body should yield 202 response [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:24

      Expected
          <int>: 204
      to equal
          <int>: 202

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:37
------------------------------
•! Panic [0.000 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Streamed
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:23
      PUT request to session URL with digest should yield 201 response [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:41

      Test Panicked
      runtime error: invalid memory address or nil pointer dereference
      /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:260

      Full Stack Trace
      github.com/bloodorangeio/reggie.(*Response).GetRelativeLocation(0xc000552000?)
        /home/ktarplee/go/pkg/mod/github.com/bloodorangeio/reggie@v0.6.1/response.go:22 +0x29
      github.com/opencontainers/distribution-spec/conformance.glob..func2.1.2.2()
        /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:43 +0x4f
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00033a278?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:113 +0xb1
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xa61537?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:64 +0xf8
      github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x8d3840?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/it_node.go:26 +0x7b
      github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0002f93b0, 0xc0001b19c8?, {0x9c6580, 0xc0002e2800})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:215 +0x2a9
      github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0002f93b0, {0x9c6580, 0xc0002e2800})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:138 +0xf2
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0002c4000, 0xc0002f93b0)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:200 +0xf1
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0002c4000)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:170 +0x1b1
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0002c4000)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:66 +0xc5
      github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0002e5110, {0x7fc02959ac30, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc00032c5d0, 0x3, 0x3}, {0x9caa38, 0xc0002e2800}, ...)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/suite/suite.go:79 +0x5e5
      github.com/onsi/ginkgo.runSpecsWithCustomReporters({0x9c6d00?, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc0002e3480, 0x3, 0x901f6c?})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:245 +0x213
      github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters({0x9c6d00, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc0000cdf40, 0x2, 0x2})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:228 +0x1a9
      github.com/opencontainers/distribution-spec/conformance.TestConformance(0x0?)
        /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:21 +0x170
      testing.tRunner(0xc00018d6c0, 0x9501c8)
        /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1576 +0x10b
      created by testing.(*T).Run
        /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1629 +0x3ea
------------------------------
•
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Monolithic
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:55
      POST request with digest and blob should yield a 201 or 202 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:65

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:75
------------------------------
•! Panic [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Monolithic
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:55
      GET request to blob URL from prior request should yield 200 or 404 based on response code [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:83

      Test Panicked
      runtime error: invalid memory address or nil pointer dereference
      /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:260

      Full Stack Trace
      github.com/opencontainers/distribution-spec/conformance.glob..func2.1.3.3()
        /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:88 +0x121
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).runSync(0xc00033a278?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:113 +0xb1
      github.com/onsi/ginkgo/internal/leafnodes.(*runner).run(0xa61537?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/runner.go:64 +0xf8
      github.com/onsi/ginkgo/internal/leafnodes.(*ItNode).Run(0x8d3840?)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/leafnodes/it_node.go:26 +0x7b
      github.com/onsi/ginkgo/internal/spec.(*Spec).runSample(0xc0002f9680, 0xc0001b19c8?, {0x9c6580, 0xc0002e2800})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:215 +0x2a9
      github.com/onsi/ginkgo/internal/spec.(*Spec).Run(0xc0002f9680, {0x9c6580, 0xc0002e2800})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/spec/spec.go:138 +0xf2
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpec(0xc0002c4000, 0xc0002f9680)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:200 +0xf1
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).runSpecs(0xc0002c4000)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:170 +0x1b1
      github.com/onsi/ginkgo/internal/specrunner.(*SpecRunner).Run(0xc0002c4000)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/specrunner/spec_runner.go:66 +0xc5
      github.com/onsi/ginkgo/internal/suite.(*Suite).Run(0xc0002e5110, {0x7fc02959ac30, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc00032c5d0, 0x3, 0x3}, {0x9caa38, 0xc0002e2800}, ...)
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/internal/suite/suite.go:79 +0x5e5
      github.com/onsi/ginkgo.runSpecsWithCustomReporters({0x9c6d00?, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc0002e3480, 0x3, 0x901f6c?})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:245 +0x213
      github.com/onsi/ginkgo.RunSpecsWithDefaultAndCustomReporters({0x9c6d00, 0xc00018d6c0}, {0x9089da, 0x22}, {0xc0000cdf40, 0x2, 0x2})
        /home/ktarplee/go/pkg/mod/github.com/onsi/ginkgo@v1.16.5/ginkgo_dsl.go:228 +0x1a9
      github.com/opencontainers/distribution-spec/conformance.TestConformance(0x0?)
        /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:21 +0x170
      testing.tRunner(0xc00018d6c0, 0x9501c8)
        /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1576 +0x10b
      created by testing.(*T).Run
        /home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/testing/testing.go:1629 +0x3ea
------------------------------
•
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Monolithic
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:55
      PUT upload of a blob should yield a 201 Response [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:104

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:114
------------------------------
•
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Monolithic
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:55
      PUT upload of a layer blob should yield a 201 Response [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:126

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:139
------------------------------
••
------------------------------
• Failure [0.002 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Chunked
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:152
      PATCH request with first chunk should return 202 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:182

      Expected
          <int>: 204
      to equal
          <int>: 202

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:199
------------------------------
•
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Chunked
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:152
      Get on stale blob upload should return 204 with a range and location [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:216

      Expected
          <int>: 400
      to equal
          <int>: 204

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:221
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Chunked
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:152
      PATCH request with second chunk should return 202 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:227

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:237
------------------------------
• Failure [0.002 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Blob Upload Chunked
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:152
      PUT request with digest should return 201 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:242

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:251
------------------------------
•
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Cross-Repository Blob Mount
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:256
      POST request to mount another repository's blob should return 201 or 202 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:268

      Expected
          <string>: /v2/myorg/other/blobs/uploads/8075872771705408622
      to equal
          <string>: /v2/myorg/other/blobs/sha256:63799bf8686261b138b932a2afc6d5921ecdba840d6249fe4f8d737385fbb67b

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:280
------------------------------
S [SKIPPING] [0.000 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Cross-Repository Blob Mount
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:256
      GET request to test digest within cross-mount namespace should return 200 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:285

      you have skipped this test.

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/setup.go:345
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Cross-Repository Blob Mount
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:256
      Cross-mounting of nonexistent blob should yield session id [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:295

      Expected
          <string>: /v2/myorg/myrepo/blobs/uploads/8282836333429469738
      to have prefix
          <string>: /v2/myorg/other/blobs/uploads/

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:299
------------------------------
S [SKIPPING] [0.000 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Cross-Repository Blob Mount
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:256
      Cross-mounting without from, and automatic content discovery enabled should return a 201 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:302

      you have skipped this test.

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/setup.go:345
------------------------------
S [SKIPPING] [0.000 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Cross-Repository Blob Mount
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:256
      Cross-mounting without from, and automatic content discovery disabled should return a 202 [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:316

      you have skipped this test.

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/setup.go:345
------------------------------
•
------------------------------
• Failure [0.002 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Manifest Upload
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:331
      PUT should accept a manifest upload [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:341

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:352
------------------------------
• Failure [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Push
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:15
    Manifest Upload
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:331
      Registry should accept a manifest upload with no layers [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:357

      Expected
          <string>: 
      not to be empty

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:367
------------------------------
•••••••
------------------------------
S [SKIPPING] [0.001 seconds]
OCI Distribution Conformance Tests
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/00_conformance_suite_test.go:12
  Content Discovery
  /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/03_discovery_test.go:16
    Setup
    /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/03_discovery_test.go:21
      Populate registry with test tags (no push) [It]
      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/03_discovery_test.go:80

      you have skipped this test.

      /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/setup.go:351
------------------------------
••••••••••••••••HTML report was created: /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/report.html
JUnit report was created: /home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/junit.xml


Summarizing 15 Failures:

[Fail] OCI Distribution Conformance Tests Pull Error codes [It] 400 response body should contain OCI-conforming JSON message 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/01_pull_test.go:205

[Fail] OCI Distribution Conformance Tests Push Blob Upload Streamed [It] PATCH request with blob in body should yield 202 response 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:37

[Panic!] OCI Distribution Conformance Tests Push Blob Upload Streamed [It] PUT request to session URL with digest should yield 201 response 
/home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:260

[Fail] OCI Distribution Conformance Tests Push Blob Upload Monolithic [It] POST request with digest and blob should yield a 201 or 202 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:75

[Panic!] OCI Distribution Conformance Tests Push Blob Upload Monolithic [It] GET request to blob URL from prior request should yield 200 or 404 based on response code 
/home/linuxbrew/.linuxbrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:260

[Fail] OCI Distribution Conformance Tests Push Blob Upload Monolithic [It] PUT upload of a blob should yield a 201 Response 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:114

[Fail] OCI Distribution Conformance Tests Push Blob Upload Monolithic [It] PUT upload of a layer blob should yield a 201 Response 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:139

[Fail] OCI Distribution Conformance Tests Push Blob Upload Chunked [It] PATCH request with first chunk should return 202 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:199

[Fail] OCI Distribution Conformance Tests Push Blob Upload Chunked [It] Get on stale blob upload should return 204 with a range and location 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:221

[Fail] OCI Distribution Conformance Tests Push Blob Upload Chunked [It] PATCH request with second chunk should return 202 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:237

[Fail] OCI Distribution Conformance Tests Push Blob Upload Chunked [It] PUT request with digest should return 201 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:251

[Fail] OCI Distribution Conformance Tests Push Cross-Repository Blob Mount [It] POST request to mount another repository's blob should return 201 or 202 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:280

[Fail] OCI Distribution Conformance Tests Push Cross-Repository Blob Mount [It] Cross-mounting of nonexistent blob should yield session id 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:299

[Fail] OCI Distribution Conformance Tests Push Manifest Upload [It] PUT should accept a manifest upload 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:352

[Fail] OCI Distribution Conformance Tests Push Manifest Upload [It] Registry should accept a manifest upload with no layers 
/home/ktarplee/code/github.com/opencontainers/distribution-spec/conformance/02_push_test.go:367

Ran 63 of 68 Specs in 0.057 seconds
FAIL! -- 48 Passed | 15 Failed | 0 Pending | 5 Skipped

--- FAIL: TestConformance (0.11s)
FAIL

It looks like you can run them with docker or as a GitHub Action.
@ktarplee ktarplee added the question Further information is requested label Apr 19, 2023
@imjasonh
Copy link
Collaborator

I think this would be helpful, yes.

Both as a test here to guard against breaking changes to this code, and possibly as a test case in the OCI distribution-spec repo to guard against unexpected changes to conformance tests. They recently started running tests against zot to help identify conformance test changes, but I think this implementation is a bit simpler to operate in a test environment than zot, since that's what it was designed for.

@jonjohnsonjr
Copy link
Collaborator

Agree it would be great.

@github-actions
Copy link

This issue is stale because it has been open for 90 days with no
activity. It will automatically close after 30 more days of
inactivity. Keep fresh with the 'lifecycle/frozen' label.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/stale question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@imjasonh @ktarplee @jonjohnsonjr