Harbor Replication Issue: Invalid Image Manifest Causes Transfer Failure and Vulnerability Scan Error #18813

Moep90 · 2023-06-12T15:17:47Z

Expected behavior and actual behavior:
Since I use harbor I replicate (cache) my images into our registry and mirror them whereever I need them.
Since OCI there seem to be some issues rising atm.

Steps to reproduce the problem:
Sync OCI any linkerd image from current stable or quay.io/jetstack/trust-manager:v0.5.0 into your registry using a replica job.

Versions:
Version

harbor version: v2.8.2-d4c34dcc
docker engine version: 24.0.2
docker-compose version: v2.18.1

Additional context:
Replication into the registry:

Job succeeded

Replicate out of the registry:

2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: docker-registry, URL: https://swr.eu-de.otc.t-systems.com, insecure: false] created
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying cache-common/cert-manager/trust-manager:[v0.5.0](source registry) to [...]/cert-manager/trust-manager:[v0.5.0](destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/cert-manager/trust-manager:v0.5.0(source registry) to [...]/cert-manager/trust-manager:v0.5.0(destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/cert-manager/trust-manager:v0.5.0 ...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact cache-common/cert-manager/trust-manager:v0.5.0 pulled
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(source registry) to [...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(destination registry)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact cache-common/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a pulled
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe(the 1th running)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe already exists on the destination registry, skip
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe completed
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202(the 1th running)...
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:339]: the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202 already exists on the destination registry, skip
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202 completed
2023-06-12T15:10:35Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact ops-[...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:504]: failed to push manifest of artifact ops-[...]/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a: http status code: 400, body: {"errors":[{"code":"MANIFEST_INVALID","message":"Invalid image, fail to parse 'manifest.json'"}]}
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 400, body: {"errors":[{"code":"MANIFEST_INVALID","message":"Invalid image, fail to parse 'manifest.json'"}]}
2023-06-12T15:10:35Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure

Trivy logs

2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:387]: {
  "uuid": "932f8b23-87a8-11eb-a090-0242ac1a0004",
  "name": "Trivy",
  "description": "The Trivy scanner adapter",
  "url": "http://trivy-adapter:8080",
  "disabled": false,
  "is_default": true,
  "health": "healthy",
  "auth": "",
  "access_credential": "[HIDDEN]",
  "skip_certVerify": false,
  "use_internal_addr": true,
  "adapter": "Trivy",
  "vendor": "Aqua Security",
  "version": "v0.42.0",
  "create_time": "2021-03-18T05:12:45.855453Z",
  "update_time": "2021-03-18T05:12:45.855455Z"
}
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:387]: {
  "registry": {
    "url": "http://core:8080",
    "authorization": "[HIDDEN]"
  },
  "artifact": {
    "namespace_id": 3211,
    "repository": "cache-common/cert-manager/trust-manager",
    "tag": "",
    "digest": "sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358",
    "mime_type": "application/vnd.oci.image.manifest.v1+json"
  }
}
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:167]: Report mime types: [application/vnd.security.vulnerability.report; version=1.1]
2023-06-12T15:02:40Z [INFO] [/pkg/scan/job.go:224]: Get report for mime type: application/vnd.security.vulnerability.report; version=1.1
2023-06-12T15:02:42Z [INFO] [/pkg/scan/job.go:245]: Report with mime type application/vnd.security.vulnerability.report; version=1.1 is not ready yet, retry after 5 seconds
2023-06-12T15:02:47Z [ERROR] [/pkg/scan/job.go:294]: check scan report with mime type application/vnd.security.vulnerability.report; version=1.1: running trivy wrapper: running trivy: exit status 1: 2023-06-12T15:02:42.365Z	�[34mINFO�[0m	Vulnerability scanning is enabled
2023-06-12T15:02:42.552Z	�[31mFATAL�[0m	image scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964): walk error: failed to extract the archive: archive/tar: invalid tar header
: general response handler: unexpected status code: 500, expected: 200

The OCI in the registry looks like so

See issue on OCI opencontainers/image-spec#1025

The text was updated successfully, but these errors were encountered:

lengrongfu · 2023-06-14T03:14:15Z

you can look this layer 80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964 length whether is 0.

Moep90 · 2023-06-14T05:33:22Z

@lengrongfu
In Quay.io it looks similar:

then in Harbor:

Just the replication/trivy cant work with it.

wy65701436 · 2023-06-19T03:37:02Z

could you please share the manifest json of ops-nexboard/cert-manager/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a? It appears that the format doesn't confirm to the OCI image spec.

Moep90 · 2023-06-19T03:48:38Z

Yeah I mean, it seams to expect more architectures then the OCI artifact has.
This seams to be an issue with the attestation-manifest-descriptor https://docs.docker.com/build/attestations/attestation-storage/#attestation-manifest-descriptor.

$ docker manifest inspect [...]/cert-manager/trust-manager:v0.5.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

Moep90 · 2023-06-23T07:19:04Z

@wy65701436 @lengrongfu Any noticable Updates on this?
More and more replication jobs are failing because of this issue.

chlins · 2023-06-24T01:02:59Z

Yeah I mean, it seams to expect more architectures then the OCI artifact has. This seams to be an issue with the attestation-manifest-descriptor https://docs.docker.com/build/attestations/attestation-storage/#attestation-manifest-descriptor.

$ docker manifest inspect [...]/cert-manager/trust-manager:v0.5.0
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v7"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 482,
         "digest": "sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "size": 566,
         "digest": "sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a",
         "platform": {
            "architecture": "unknown",
            "os": "unknown"
         }
      }
   ]
}

This is the manifest list for the cert-manager/trust-manager:v0.5.0, could you check the manifest of one arch such as amd64 and unknown? The manifest can also be retrieved by Harbor API. curl -u admin:xxxxx https://harbor.doamin/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a

Moep90 · 2023-07-04T15:37:45Z

@chlins

curl -u admin:xxxxx https://harbor.doamin/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a

Returns:

{
  "errors": [
    {
      "code": "MANIFEST_UNKNOWN",
      "message": "OCI manifest found, but accept header does not support OCI manifests"
    }
  ]
}

Moep90 · 2023-07-06T03:05:00Z

@wy65701436 @lengrongfu @lengrongfu any progress?
We're seeing this with more and more images every week.
Now since more and more projects are adopting OCI.
Replication in my company is a crucial feature.

Moep90 · 2023-07-10T15:46:48Z

@chlins any update? As mentioned prior, we're blocked by this.
It would be great to get an estimate.

chlins · 2023-07-11T00:41:30Z

@chlins

curl -u admin:xxxxx https://harbor.doamin/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a

Returns:
{
  "errors": [
    {
      "code": "MANIFEST_UNKNOWN",
      "message": "OCI manifest found, but accept header does not support OCI manifests"
    }
  ]
}

Please try to add the Accept header when curl request, the value can refer to https://docs.docker.com/registry/spec/manifest-v2-2/#media-types.
e.g curl -u admin:xxxxx -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://harbor.doamin/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a

chlins · 2023-07-11T00:54:25Z

I tried to replicate the image you mentioned quay.io/jetstack/trust-manager:v0.5.0 in my environment, eventually the task has been finished successfully.

The replication logs:

2023-07-11T00:46:37Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: quay, URL: https://quay.io, insecure: true] created
2023-07-11T00:46:37Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: harbor, URL: http://core:8080, insecure: true] created
2023-07-11T00:46:37Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying jetstack/trust-manager:[v0.5.0](source registry) to jetstack/trust-manager:[v0.5.0](destination registry)...
2023-07-11T00:46:37Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:v0.5.0(source registry) to jetstack/trust-manager:v0.5.0(destination registry)...
2023-07-11T00:46:37Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:v0.5.0 ...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:v0.5.0 pulled
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(source registry) to jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(destination registry)...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a pulled
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe(the 1th running)...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6be20a52f0e9ea097d4e334ca45ef7ad61b316374bb5aeff8fda9c5fe1b2a7fe completed
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202(the 1th running)...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:de11bf62c62d8e59c187692126a4f833a8647d1f1cfd70deecd50305b5a20202 completed
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a ...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a pushed
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(source registry) to jetstack/trust-manager:sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a(destination registry) completed
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628(source registry) to jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628(destination registry)...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628 ...
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628 pulled
2023-07-11T00:46:38Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:ff1a4d800eea6cbfd0a5349f1044d6838420985beda8f6c1bc7d91c30ca0bf47(the 1th running)...
2023-07-11T00:46:39Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:ff1a4d800eea6cbfd0a5349f1044d6838420985beda8f6c1bc7d91c30ca0bf47 completed
2023-07-11T00:46:39Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:314d6a22a3212566ff1fa9a84d9ec6ccc9d3a40ae6b5965f3b3a0a3571673418(the 1th running)...
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:314d6a22a3212566ff1fa9a84d9ec6ccc9d3a40ae6b5965f3b3a0a3571673418 completed
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628 ...
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628 pushed
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628(source registry) to jetstack/trust-manager:sha256:d657adf096d6c71799c6e59b6902f2b8912a4b2c0b23435c4152c44be59e7628(destination registry) completed
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574(source registry) to jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574(destination registry)...
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574 ...
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574 pulled
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:68e663a06a01ba097a2651a9446d0a41931b20ac39a9e668c6ee84a55b7b70de(the 1th running)...
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:68e663a06a01ba097a2651a9446d0a41931b20ac39a9e668c6ee84a55b7b70de completed
2023-07-11T00:46:40Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:42bde91621bc704e17ef4fa3fe26ffc5352e39701b0a9ada8ab253c5f5ebbf08(the 1th running)...
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:42bde91621bc704e17ef4fa3fe26ffc5352e39701b0a9ada8ab253c5f5ebbf08 completed
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574 ...
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574 pushed
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574(source registry) to jetstack/trust-manager:sha256:3ddede62efc476e1450eff39a2e6ba90d37bb9276d92c39c4b775edc47ed2574(destination registry) completed
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33(source registry) to jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33(destination registry)...
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33 ...
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33 pulled
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:398f8c671b036c3160ef9f4899e3e0a25573992177f51229a9e8d38fc7b538ed(the 1th running)...
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:398f8c671b036c3160ef9f4899e3e0a25573992177f51229a9e8d38fc7b538ed completed
2023-07-11T00:46:41Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:86f04a70a85f10807521f8647e1207c61ea78c9a5008e352119ce146ff465fac(the 1th running)...
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:86f04a70a85f10807521f8647e1207c61ea78c9a5008e352119ce146ff465fac completed
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33 ...
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33 pushed
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33(source registry) to jetstack/trust-manager:sha256:1c5bafda555ebc6bc8769890fcae82c8a901127f948486e4bd95b3aed0ecae33(destination registry) completed
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358(source registry) to jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358(destination registry)...
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358 ...
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358 pulled
2023-07-11T00:46:42Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6ddd1691a4cc2cf22f77434879f3b07450315e5574002b993c3d7b4471bd1d7d(the 1th running)...
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6ddd1691a4cc2cf22f77434879f3b07450315e5574002b993c3d7b4471bd1d7d completed
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964(the 1th running)...
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964 completed
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358 ...
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358 pushed
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358(source registry) to jetstack/trust-manager:sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358(destination registry) completed
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4(source registry) to jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4(destination registry)...
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4 ...
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4 pulled
2023-07-11T00:46:43Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:81761aef4e4372a7863c8ad48c0f41af152c31785131810868e8d66d131ae189(the 1th running)...
2023-07-11T00:46:44Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:81761aef4e4372a7863c8ad48c0f41af152c31785131810868e8d66d131ae189 completed
2023-07-11T00:46:44Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:eaf11e437ae8cfa1a67476ab392e71c51690c5b9bc9af45d0fbe1ced1d3ffdf3(the 1th running)...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:eaf11e437ae8cfa1a67476ab392e71c51690c5b9bc9af45d0fbe1ced1d3ffdf3 completed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4 ...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4 pushed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4(source registry) to jetstack/trust-manager:sha256:b4fe15fe560385db3bb53d3c26742a4323cfa4cfeeda9595f8a522a2159924f4(destination registry) completed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4(source registry) to jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4(destination registry)...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4 ...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4 pulled
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:111ce1977414d3ed878c2e7d39b58d7e737e79c136d2ec565aa2f44f97bc56a1(the 1th running)...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:111ce1977414d3ed878c2e7d39b58d7e737e79c136d2ec565aa2f44f97bc56a1 completed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:deff6afbebfc77c5148314d5102db71ff1f614397254d9871b2f970a0c29c31f(the 1th running)...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:deff6afbebfc77c5148314d5102db71ff1f614397254d9871b2f970a0c29c31f completed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4 ...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4 pushed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4(source registry) to jetstack/trust-manager:sha256:82acbf9899acbfd4f5842ebb99dbdb1686c38a8823518e6ed3df8dcb6e9b68e4(destination registry) completed
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a(source registry) to jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a(destination registry)...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a ...
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:473]: the manifest of artifact jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a pulled
2023-07-11T00:46:45Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:6322ac0adbe6706aaf40781d71bd58270d87cebe1d0302cbb4876fcd04857c00(the 1th running)...
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:6322ac0adbe6706aaf40781d71bd58270d87cebe1d0302cbb4876fcd04857c00 completed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:289]: copying the blob sha256:87e5fc8f97175f997a4cef4ff6d818d2252b9b96e7417346a54ae77f094511bc(the 1th running)...
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:291]: copy the blob sha256:87e5fc8f97175f997a4cef4ff6d818d2252b9b96e7417346a54ae77f094511bc completed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a ...
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a pushed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a(source registry) to jetstack/trust-manager:sha256:1b82e70992869227dae9d8d4fb92d0f7ff1af626f7140deb9518cdf3863ba28a(destination registry) completed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:496]: pushing the manifest of artifact jetstack/trust-manager:v0.5.0 ...
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:508]: the manifest of artifact jetstack/trust-manager:v0.5.0 pushed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:253]: copy jetstack/trust-manager:v0.5.0(source registry) to jetstack/trust-manager:v0.5.0(destination registry) completed
2023-07-11T00:46:46Z [INFO] [/controller/replication/transfer/image/transfer.go:204]: copy jetstack/trust-manager:[v0.5.0](source registry) to jetstack/trust-manager:[v0.5.0](destination registry) completed

Moep90 · 2023-07-11T05:08:51Z

@chlins

curl -u admin:xxxxx -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://harbor.doamin/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a

$ curl -u admin:xxxxxx -H "Accept: application/vnd.docker.distribution.manifest.v2+json"  https://harbor.example.com/v2/cache-common/cert-manager/trust-manager/manifests/sha256:d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   122  100   122    0     0   2033      0 --:--:-- --:--:-- --:--:--  2033
{
  "errors": [
    {
      "code": "MANIFEST_UNKNOWN",
      "message": "OCI manifest found, but accept header does not support OCI manifests"
    }
  ]
}

I tried to replicate the image you mentioned quay.io/jetstack/trust-manager:v0.5.0 in my environment, eventually the task has been finished successfully.

Well yes, you did a replication from quay.io to your registry which works fine for me as well.
What doesn't work is the fact that I try to replicate from harbor.example.com TO another registry. It seams to try to replicate the whole OCI artifact which does not contain certain types of architectures....

chlins · 2023-07-11T06:44:12Z

OK, so the image can be replicated from quay.io to harbor successfully, but can not again replicate the image to the remote registry, which vendor of your target registry? I've tried to replicate the image from harbor to harbor, it works normal.

Moep90 · 2023-07-11T06:46:13Z

The vender is huawei SWR or Software Repository for Container.

chlins · 2023-07-11T06:53:14Z

From my perspective, the issue may relate with remote registry because from the replication logs it shows the process is normal, and eventually it failed due to receive the error from remote registry. You can try to replicate the image to other registries such as DockerHub for verification.

Moep90 · 2023-07-11T07:02:11Z

@chlins Could you also try with this image cr.l5d.io/linkerd/policy-controller:stable-2.13.1?
Pull: Success
Push: Failed

2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: docker-registry, URL: https://swr.eu-de.otc.t-systems.com, insecure: false] created
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying cache-common/linkerd/policy-controller:[stable-2.13.1](source registry) to ops-luca/linkerd/policy-controller:[stable-2.13.1](destination registry)...
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/linkerd/policy-controller:stable-2.13.1(source registry) to ops-luca/linkerd/policy-controller:stable-2.13.1(destination registry)...
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/linkerd/policy-controller:stable-2.13.1 ...
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:470]: failed to pull the manifest of artifact cache-common/linkerd/policy-controller:stable-2.13.1: http status code: 404, body: {"errors":[{"code":"NOT_FOUND","message":"artifact cache-common/linkerd/policy-controller:stable-2.13.1 not found"}]}
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 404, body: {"errors":[{"code":"NOT_FOUND","message":"artifact cache-common/linkerd/policy-controller:stable-2.13.1 not found"}]}
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure

chlins · 2023-07-11T07:09:40Z

@chlins Could you also try with this image cr.l5d.io/linkerd/policy-controller:stable-2.13.1? Pull: Success Push: Failed

2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:139]: client for source registry [type: harbor, URL: http://core:8080, insecure: true] created
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:149]: client for destination registry [type: docker-registry, URL: https://swr.eu-de.otc.t-systems.com, insecure: false] created
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:182]: copying cache-common/linkerd/policy-controller:[stable-2.13.1](source registry) to ops-luca/linkerd/policy-controller:[stable-2.13.1](destination registry)...
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:210]: copying cache-common/linkerd/policy-controller:stable-2.13.1(source registry) to ops-luca/linkerd/policy-controller:stable-2.13.1(destination registry)...
2023-07-11T06:02:55Z [INFO] [/controller/replication/transfer/image/transfer.go:467]: pulling the manifest of artifact cache-common/linkerd/policy-controller:stable-2.13.1 ...
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:470]: failed to pull the manifest of artifact cache-common/linkerd/policy-controller:stable-2.13.1: http status code: 404, body: {"errors":[{"code":"NOT_FOUND","message":"artifact cache-common/linkerd/policy-controller:stable-2.13.1 not found"}]}
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:194]: http status code: 404, body: {"errors":[{"code":"NOT_FOUND","message":"artifact cache-common/linkerd/policy-controller:stable-2.13.1 not found"}]}
2023-07-11T06:02:55Z [ERROR] [/controller/replication/transfer/image/transfer.go:200]: got error during the whole transfer period, mark the job failure

Please check whether the artifact cache-common/linkerd/policy-controller:stable-2.13.1 existed on your harbor.

Moep90 · 2023-07-11T07:17:38Z

@chlins it does, like I said, the pull to cache-common works perfectly.

Moep90 · 2023-07-11T07:24:22Z

@chlins
Trivy adapter reports:
FYI - we had Trivy to enable scan images on push enabled

Jul 11 07:23:46 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:a3618870e87338ee7161cfad303907055b242560b373c545dd8d55dc57dad883","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:46Z"}
Jul 11 07:23:46 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:46Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"8f8c0c0b4318883e4e2e84d8","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:583bdaf4c614da89a2f806ce2d04071b36ad532d25057f056e7e32753153c63b","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:51 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:ad8554f1164af23bed67367c4b1c801706f35e089e54b4d1d7d895fd3df76989","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:51Z"}
Jul 11 07:23:51 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:51Z"}
Jul 11 07:23:52 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"7e587b554fe9fc291ddf1a08","time":"2023-07-11T07:23:52Z"}
Jul 11 07:23:52 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"48f24baa26e1a32749e63898","time":"2023-07-11T07:23:52Z"}

Moep90 · 2023-07-11T07:28:41Z

@chlins I assume trivy uses the wrong mime_type, could that be?

  "artifact": {
    "namespace_id": 3211,
    "repository": "cache-common/linkerd/policy-controller",
    "tag": "",
    "digest": "sha256:583bdaf4c614da89a2f806ce2d04071b36ad532d25057f056e7e32753153c63b",
    "mime_type": "application/vnd.oci.image.manifest.v1+json" <------ see here
  }

chlins · 2023-07-11T07:34:11Z

cr.l5d.io/linkerd/policy-controller:stable-2.13.1

This image can also be replicated successfully in my environment...

chlins · 2023-07-11T07:37:19Z

@zyyw Could you help to confirm the supported list of layer type with trivy?

Moep90 · 2023-07-28T04:05:57Z

@chlins / @zyyw any update?

W4SD · 2023-08-01T14:32:20Z

Related?
#18186

Moep90 · 2023-08-04T12:47:03Z

@chlins / @zyyw / @lengrongfu / @wy65701436 any update?

github-actions · 2023-10-04T09:03:37Z

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions · 2023-11-03T09:03:53Z

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

sizowie · 2023-11-03T09:10:53Z

Please do not close.

Moep90 · 2023-11-03T09:18:37Z

as well as respond please @chlins / @zyyw / @lengrongfu / @wy65701436 any update?

zyyw · 2023-11-10T08:23:24Z

@chlins Trivy adapter reports: FYI - we had Trivy to enable scan images on push enabled

Jul 11 07:23:46 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:a3618870e87338ee7161cfad303907055b242560b373c545dd8d55dc57dad883","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:46Z"}
Jul 11 07:23:46 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:46Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:46.354Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:46.517Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:dfcfb662036ae764fb014ea3d7860cff76dc6ce5dfc791eba7d0b9a8fb130a5f): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"8f8c0c0b4318883e4e2e84d8","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:583bdaf4c614da89a2f806ce2d04071b36ad532d25057f056e7e32753153c63b","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:47 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:47Z"}
Jul 11 07:23:51 172.22.0.1 trivy-adapter[915393]: {"exit_code":1,"image_ref":"core:8080/cache-common/linkerd/policy-controller@sha256:ad8554f1164af23bed67367c4b1c801706f35e089e54b4d1d7d895fd3df76989","level":"error","msg":"Running trivy failed","std_out":"2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","time":"2023-07-11T07:23:51Z"}
Jul 11 07:23:51 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan failed","time":"2023-07-11T07:23:51Z"}
Jul 11 07:23:52 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:47.465Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:47.604Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:3ddc9b908d45e90de698019ee524575cec4b083f7e3c7c29065a96591706c44c): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"7e587b554fe9fc291ddf1a08","time":"2023-07-11T07:23:52Z"}
Jul 11 07:23:52 172.22.0.1 trivy-adapter[915393]: {"error":"running trivy wrapper: running trivy: exit status 1: 2023-07-11T07:23:51.767Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2023-07-11T07:23:51.915Z\t\u001b[31mFATAL\u001b[0m\timage scan error: scan error: scan failed: failed analysis: analyze error: pipeline error: failed to analyze layer (sha256:6b56408e875569227c049bfe3c3640266c8bc73fc4f4ecb61543e212b2e40ad4): walk error: failed to extract the archive: archive/tar: invalid tar header\n","level":"error","msg":"Scan job failed","scan_job_id":"48f24baa26e1a32749e63898","time":"2023-07-11T07:23:52Z"}

Sorry for the delayed response. We did reproduce this error and will discuss it with the Trivy team. This error message is returned by Trivy:

failed to extract the archive

Step to reproduce:

deploy a Harbor instance, replicate quay.io/jetstack/trust-manager:v0.5.0 to Harbor
scan the artifact

Trivy scan log after click the View Log link.

zyyw · 2023-11-10T09:01:00Z

Artifacts that are scanned successfully have application/vnd.oci.image.layer.v1.tar+gzip for layers.mediaType:

sha256:d9fb9662
sha256:d657adf0
sha256:3ddede62
sha256:1c5bafda
https://gist.github.com/zyyw/1732ae22b3a66938ef7c18b4892d934b

while those errored out by Trivy doesn't have application/vnd.oci.image.layer.v1.tar+gzip for layers.mediaType, instead it's application/vnd.in-toto+json:

sha256:06c88ccf
sha256:b4fe15fe
sha256:82acbf98
sha256:1b82e709
https://gist.github.com/zyyw/e80c93100ac316c47426d6ba0f14239a

@knqyf263
It seem that when Trivy is trying to scan the artifact, it inspects layers and expects it to be a tar layer:

https://github.com/aquasecurity/trivy/blob/cb241a800774a37527566dd3895575b935003f2b/pkg/fanal/artifact/image/image.go#L281

https://github.com/aquasecurity/trivy/blob/cb241a800774a37527566dd3895575b935003f2b/pkg/fanal/walker/tar.go#L48

knqyf263 · 2023-11-13T02:32:33Z

It is not a container image. It makes sense to fail to scan it. What does Harbor expect in this case? IMO, Harbor should not show the "Scan" button and trigger scanning on the provenance. It is not only provenance technically. The scanning should not be triggered on anything other than supported types, images (and SBOM). What do you think?

$ crane manifest quay.io/jetstack/trust-manager@sha256:06c88ccf61e2d5f1c6beca7feca4b12c9b69a37ed7eb27dedc0a1a6db392d358
{
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:6ddd1691a4cc2cf22f77434879f3b07450315e5574002b993c3d7b4471bd1d7d",
    "size": 167
  },
  "layers": [
    {
      "mediaType": "application/vnd.in-toto+json",
      "digest": "sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964",
      "size": 1147,
      "annotations": {
        "in-toto.io/predicate-type": "https://slsa.dev/provenance/v0.2"
      }
    }
  ]
}

$ crane blob quay.io/jetstack/trust-manager@sha256:80b00ffd2d00d0c46ed7ad0a13877402398d0c3a9617423d6e4694c7bfed5964 | jq .
{
  "_type": "https://in-toto.io/Statement/v0.1",
  "predicateType": "https://slsa.dev/provenance/v0.2",
  "subject": [
    {
      "name": "pkg:docker/quay.io/jetstack/trust-manager@v0.5.0?platform=linux%2Famd64",
      "digest": {
        "sha256": "d9fb966245a7fa6e59868d32ac9d5dac4f2ad92ac2982ed5f31ee7320a36552a"
      }
    }
  ],
  "predicate": {
    "builder": {
      "id": ""
    },
    "buildType": "https://mobyproject.org/buildkit@v1",
    "materials": [
      {
        "uri": "pkg:docker/golang@1.19?platform=linux%2Famd64",
        "digest": {
          "sha256": "9613596d7405705447f36440a59a3a2a1d22384c7568ae1838d0129964c5ba13"
        }
      }
    ],
    "invocation": {
      "configSource": {
        "entryPoint": "Dockerfile"
      },
      "parameters": {
        "frontend": "dockerfile.v0",
        "locals": [
          {
            "name": "context"
          },
          {
            "name": "dockerfile"
          }
        ]
      },
      "environment": {
        "platform": "linux/amd64"
      }
    },
    "metadata": {
      "buildInvocationID": "sdjwl6omebhb1sqxmrw6w93ug",
      "buildStartedOn": "2023-05-19T10:55:50.912447143Z",
      "buildFinishedOn": "2023-05-19T11:19:36.090418153Z",
      "completeness": {
        "parameters": false,
        "environment": true,
        "materials": false
      },
      "reproducible": false,
      "https://mobyproject.org/buildkit@v1#metadata": {
        "vcs": {
          "revision": "858eced8cb85116ebb406ebd71e238df8c37ed91",
          "source": "git@github.com:SgtCoDFish/cert-manager-trust-manager.git"
        }
      }
    }
  }
}

knqyf263 · 2023-11-13T02:36:04Z

The capabilities may help.
https://github.com/goharbor/pluggable-scanner-spec/blob/89e8adcd4c9b30fdd12394fb4aa1f27254d97a32/api/spec/scanner-adapter-openapi-v1.2.yaml#L20-L23

knqyf263 · 2023-11-13T02:40:54Z

BTW, I think the attestations should ideally be linked to the artifact via OCI reference API rather than mixed in the image manifest list.

zyyw · 2023-11-13T09:54:51Z

BTW, I think the attestations should ideally be linked to the artifact via OCI reference API rather than mixed in the image manifest list.

Yeah, I agree with you on that. And if these non-image artifacts are linked to subject artifacts via OCI reference API, they will be stored in Harbor as accessories, which will not be scanned as expected. It seems this should be the best practice, rather than putting these attestations in the image index file.

knqyf263 · 2023-11-14T02:45:38Z

Anyway, a non-image digest is passed to Trivy and the scanning failed. It's an expected behavior on the scanner side. Please let me know if you request any changes to the scanners.

zyyw · 2023-11-14T04:55:01Z

Anyway, a non-image digest is passed to Trivy and the scanning failed. It's an expected behavior on the scanner side. Please let me know if you request any changes to the scanners.

Sure, and thank you for the feedback!

ChristianCiach · 2023-11-14T13:07:08Z

For reference, this issue is essentially a duplicate of my issue at #17630

Also, shouldn't this issue be reopened? It hasn't been fixed and the issue still appears with current harbor versions.

sizowie · 2024-02-14T13:36:13Z

For reference, this issue is essentially a duplicate of my issue at #17630

Also, shouldn't this issue be reopened? It hasn't been fixed and the issue still appears with current harbor versions.

+1 - still an issue, please reopen the issue.

wy65701436 assigned chlins Jun 19, 2023

wy65701436 added the area/replication label Jun 19, 2023

github-actions bot added the Stale label Oct 4, 2023

github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Nov 3, 2023

zyyw mentioned this issue Nov 13, 2023

Images build with buildx 0.10.x with provenance enabled, harbor tries to init scan and fails on OCI manifest files #18186

Closed

Harbor Replication Issue: Invalid Image Manifest Causes Transfer Failure and Vulnerability Scan Error #18813

Harbor Replication Issue: Invalid Image Manifest Causes Transfer Failure and Vulnerability Scan Error #18813

Comments

Moep90 commented Jun 12, 2023 • edited Loading

lengrongfu commented Jun 14, 2023

Moep90 commented Jun 14, 2023 • edited Loading

wy65701436 commented Jun 19, 2023

Moep90 commented Jun 19, 2023 • edited Loading

Moep90 commented Jun 23, 2023

chlins commented Jun 24, 2023 • edited Loading

Moep90 commented Jul 4, 2023

Moep90 commented Jul 6, 2023

Moep90 commented Jul 10, 2023

chlins commented Jul 11, 2023 • edited Loading

chlins commented Jul 11, 2023

Moep90 commented Jul 11, 2023 • edited Loading

chlins commented Jul 11, 2023

Moep90 commented Jul 11, 2023

chlins commented Jul 11, 2023

Moep90 commented Jul 11, 2023 • edited Loading

chlins commented Jul 11, 2023

Moep90 commented Jul 11, 2023

Moep90 commented Jul 11, 2023 • edited Loading

Moep90 commented Jul 11, 2023

chlins commented Jul 11, 2023

chlins commented Jul 11, 2023

Moep90 commented Jul 28, 2023

W4SD commented Aug 1, 2023

Moep90 commented Aug 4, 2023 • edited Loading

github-actions bot commented Oct 4, 2023

github-actions bot commented Nov 3, 2023

sizowie commented Nov 3, 2023

Moep90 commented Nov 3, 2023 • edited Loading

zyyw commented Nov 10, 2023 • edited Loading

zyyw commented Nov 10, 2023 • edited Loading

knqyf263 commented Nov 13, 2023 • edited Loading

knqyf263 commented Nov 13, 2023

knqyf263 commented Nov 13, 2023

zyyw commented Nov 13, 2023 • edited Loading

knqyf263 commented Nov 14, 2023

zyyw commented Nov 14, 2023

ChristianCiach commented Nov 14, 2023 • edited Loading

sizowie commented Feb 14, 2024

Moep90 commented Jun 12, 2023 •

edited

Loading

Moep90 commented Jun 14, 2023 •

edited

Loading

Moep90 commented Jun 19, 2023 •

edited

Loading

chlins commented Jun 24, 2023 •

edited

Loading

chlins commented Jul 11, 2023 •

edited

Loading

Moep90 commented Jul 11, 2023 •

edited

Loading

Moep90 commented Jul 11, 2023 •

edited

Loading

Moep90 commented Jul 11, 2023 •

edited

Loading

Moep90 commented Aug 4, 2023 •

edited

Loading

Moep90 commented Nov 3, 2023 •

edited

Loading

zyyw commented Nov 10, 2023 •

edited

Loading

zyyw commented Nov 10, 2023 •

edited

Loading

knqyf263 commented Nov 13, 2023 •

edited

Loading

zyyw commented Nov 13, 2023 •

edited

Loading

ChristianCiach commented Nov 14, 2023 •

edited

Loading