feat(storage): enable parallel writes by using per-repo and per-digest locking #2968

andaaron · 2025-02-15T11:22:16Z

Should fix issues such as #2964

lock per repo on pushes/pulls/retention, in short index operations
lock per digest when using multiple operations affecting the cachedb and storage
(blob writes/deletes/moves/links in storage which need to be in accordance with cachedb content)

Do not lock multiple repos at the same time in the same goroutine! It will cause deadlocks.
Same applies to digests.

In separate commits:

show more error information in zb output
gc stress tests to save logs as artifacts

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2025-02-15T11:41:07Z

Codecov Report

Attention: Patch coverage is 91.58485% with 60 lines in your changes missing coverage. Please review.

Project coverage is 90.94%. Comparing base (d465690) to head (e8bf6e5).

Files with missing lines	Patch %	Lines
pkg/storage/imagestore/imagestore.go	92.04%	27 Missing and 8 partials ⚠️
pkg/test/oci-utils/oci_layout.go	74.00%	10 Missing and 3 partials ⚠️
pkg/meta/boltdb/boltdb.go	0.00%	3 Missing and 1 partial ⚠️
pkg/meta/dynamodb/dynamodb.go	42.85%	3 Missing and 1 partial ⚠️
pkg/meta/redis/redis.go	0.00%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2968      +/-   ##
==========================================
- Coverage   91.00%   90.94%   -0.06%     
==========================================
  Files         176      177       +1     
  Lines       32900    32972      +72     
==========================================
+ Hits        29940    29987      +47     
- Misses       2234     2254      +20     
- Partials      726      731       +5

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rchincha · 2025-02-27T07:12:15Z

Looks reasonable.

Are there actual benchmark numbers that show improvement under contention?

andaaron · 2025-02-27T14:12:05Z

Looks reasonable.

Are there actual benchmark numbers that show improvement under contention?

I have updated the tests using the benchmark HG action to post the comparison in the job summary:
https://github.com/project-zot/zot/actions/runs/13567724400?pr=2968

And:
https://github.com/project-zot/zot/actions/runs/13567724423?pr=2968

The results for minio are almost unchanged.
For local storage we see some improved performance for pushes, and some decreases for pulling (and there is some variations between runs).
I think we should see in a more realistic environment.

andaaron · 2025-02-27T21:49:19Z

I ran zb locally with config (note GC and dedupe are enabled):

{
    "distSpecVersion": "1.1.0-dev",
    "storage": {
        "rootDirectory": "/data/hdd/zot",
        "dedupe": true,
        "gc": true
    },
    "http": {
        "address": "0.0.0.0",
        "port": "8080",
        "realm": "zot",
        "tls": {
          "cert": "/data/ssd/cert/server.cert",
          "key": "/data/ssd/cert/server.key"
        }
    },
    "log": {
        "level": "debug",
        "output": "/data/hdd/zot.log"
    },
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "2h"
            }
        },
        "metrics": {
            "enable": true
        },
        "ui": {
            "enable": true
        },
        "mgmt": {
            "enable": true
        },
        "trust": {
            "enable": true,
            "cosign": true,
            "notation": true
        }
    }
}

And zb command: bin/zb-linux-amd64 -c 100 -n 1000 -o stdout https://localhost:8080

This is the result with the code in #2996 (just some ci improvements) zb_diff.md
This is the result with the code in this PR (includes lock changes) zb_locks.md
This is the diff of the files above zb_ref.md

shcherbak · 2025-03-02T17:50:00Z

#2999

linking issue with tests and comments

rchincha · 2025-03-02T19:23:13Z

We really need to add DELETEs in the zb tests, since that path will really stress test gc and dedupe.

rchincha · 2025-03-03T19:19:32Z

How does dedupe locking and imgstore locking work wrt each other?

Thread1: Create path imgstore lock, unlock, dedupe lock, unlock
Thread2: Delete path imgstore lock, unlock, dedupe lock, unlock
Thread3: Update path imgstore lock, unlock, dedupe lock, unlock

They can cross each other in unexpected ways!

Also, I am expected dedupe "lock/unlock" actually done in boltdb right?

andaaron · 2025-03-03T21:57:02Z

How does dedupe locking and imgstore locking work wrt each other?

Can you clarify what you mean? The image store has locks per repo, and per digest.
There is no such thing as a "dedupe" object comparable to an "image store" object.

Thread1: Create path imgstore lock, unlock, dedupe lock, unlock Thread2: Delete path imgstore lock, unlock, dedupe lock, unlock Thread3: Update path imgstore lock, unlock, dedupe lock, unlock

They can cross each other in unexpected ways!

Again there is no such thing as a dedupe lock.
Point me to the code where the scenario you mention happens.
The workflow (although not as clear in the code is):
a) for http requests: acquire lock on repo, do some image store ops, if dedupe logic is needed aquire lock on digest, so some more image store ops do some cachedb ops, some more images store ops, and so on, release lock on digest, release lock on repo. In some cases the lock on digest is not needed, if we don't need to dedupe.
b) for the dedupe background tasks the lock on repo are not used at all, only the locks on digests are, per digest, for both image store and cache db, after which the digest locks are released.

Also, I am expected dedupe "lock/unlock" actually done in boltdb right?

I do not know what you are talking about. boltdb has its own internal transaction/locking mechanism, which we don't meddle with.

rchincha · 2025-03-05T07:32:26Z

If you have to take two locks, make sure ordering is ensured, maybe by repo names.

andaaron · 2025-03-05T07:48:41Z

If you have to take two locks, make sure ordering is ensured, maybe by repo names.

Like I said 1st lock is on the repo, 2nd lock is on the digest. And on release digest, then repo.

We don't lock multiple repos at the same time in the same goroutine, as that results in deadlocks. Also we don't lock multiple digests at the same time in the same goroutine, that's not necessary. I had also mentioned this in #2968 (comment).

Ideally we would have locked multiple repos while dedupe ran on them, tried it, and it was 100% guarantee of a deadlock.
That's why I introduced the lock per digest.
Unfortunately not locking multiple repos at the same time also makes some of the functions returning repo lists best effort.

rchincha · 2025-03-05T09:29:50Z

I am still at the code carefully.
So 500s need to be understood still?

andaaron · 2025-03-05T11:05:13Z

We need to reproduce / fix the docker buildx issue, but I don't think it is related to this change. It was reported in 2024 as well, and it is not clear if they tested the workaround back then.

andaaron · 2025-03-06T10:45:47Z

If you have to take two locks, make sure ordering is ensured, maybe by repo names.

I would give another shot to locking multiple repos instead of using the digest locking as it is implemented right now.

The issue is we can't order the repo locks. When the user pushes a blob, that specific repo is locked right away, and only later, when checking for duplicates, we can lock the rest of the affected repos.
So we can't guarantee a consistent order of aquiring the repo locks when a blob is being pushed to multiple repos concurrently.

shcherbak · 2025-03-06T16:14:00Z

@andaaron is there any updates? I can try new build with fix, because errors are really painful (

andaaron · 2025-03-06T16:44:17Z

@andaaron is there any updates? I can try new build with fix, because errors are really painful (

Hi @shcherbak, not from my side. I will try to look into this next weekend.
While the docker buildx mediatype issue should be reproducible, it is not clear to me what is the code path returning error 500 on HEAD request, and how to reproduce that. I see a couple of log messages in the logs presented in #2999, but those are on PATCH I believe.
Can you send the actual logs from zot (zot logs, not client logs, or statistics), in which the 500 error is returned, maybe include the other log messages just before the errors?

shcherbak · 2025-03-06T16:49:34Z

@andaaron is there any updates? I can try new build with fix, because errors are really painful (

Hi @shcherbak, not from my side. I will try to look into this next weekend. While the docker buildx mediatype issue should be reproducible, it is not clear to me what is the code path returning error 500 on HEAD request, and how to reproduce that. I see a couple of log messages in the logs presented in #2999, but those are on PATCH I believe. Can you send the actual logs from zot (zot logs, not client logs, or statistics), in which the 500 error is returned, maybe include the other log messages just before the errors?

i'm on collecting surrounding information about several concrete events, will send in an hour.

shcherbak · 2025-03-06T18:47:26Z

HEAD 500 and surrounding logs (grep -C3 -A3)

{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.51:32676","method":"HEAD","path":"/v2/REPO1/blobs/sha256:c55e8b04fc29899c7f01cb628c9618be2812a692047bcff10dcde40c1567f8a3","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":7009991,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:21:41.390593206Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"KUBELETUSER","component":"session","clientIP":"10.12.18.51:32688","method":"HEAD","path":"/v2/REPO2/manifests/7c03094edb1fb7303e1e000c1f7e52fcf4de95b5","statusCode":404,"latency":"0s","bodySize":250,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*"],"Authorization":["******"],"Referer":["https://registry.smen.space/v2/REPO2/manifests/7c03094edb1fb7303e1e000c1f7e52fcf4de95b5"],"User-Agent":["containerd/v1.6.34"],"X-Forwarded-For":["10.1.0.90"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["10.1.0.90"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":7010052,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:21:44.489158632Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.51:16706","method":"HEAD","path":"/v2/REPO1/blobs/sha256:e61fe0e90e09f65c5b7bd55704c805ff57df7c54d7d4b0eab655ae3a122d6eab","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":7010116,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:21:45.746506566Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"KUBELETUSER","component":"session","clientIP":"10.12.18.51:56354","method":"HEAD","path":"/v2/REPO3/manifests/bd8ee510f18108740f3599da72c4f15cff5c1f26","statusCode":500,"latency":"0s","bodySize":151,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*"],"Authorization":["******"],"Referer":["https://registry.smen.space/v2/REPO3/manifests/bd8ee510f18108740f3599da72c4f15cff5c1f26"],"User-Agent":["containerd/v1.6.34"],"X-Forwarded-For":["10.1.0.143"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["10.1.0.143"]},"goroutine":7010060,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:21:56.866560051Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"KUBELETUSER","component":"session","clientIP":"10.12.18.51:56364","method":"HEAD","path":"/v2/REPO4/manifests/32f1ed34e5b0dc7aebb06f00ac19ecd4e17e56c9","statusCode":404,"latency":"0s","bodySize":250,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*"],"Authorization":["******"],"Referer":["https://registry.smen.space/v2/REPO4/manifests/32f1ed34e5b0dc7aebb06f00ac19ecd4e17e56c9"],"User-Agent":["containerd/v1.6.34"],"X-Forwarded-For":["10.1.0.17"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["10.1.0.17"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":7010130,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:22:02.295866587Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"KUBELETUSER","component":"session","clientIP":"10.12.18.51:44150","method":"HEAD","path":"/v2/REPO5//manifests/85ef7adcb5e4d70a483d20ac973914e630f128f9","statusCode":404,"latency":"0s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*"],"Authorization":["******"],"Referer":["https://registry.smen.space/v2/REPO5//manifests/85ef7adcb5e4d70a483d20ac973914e630f128f9"],"User-Agent":["containerd/v1.6.34"],"X-Forwarded-For":["10.1.0.143"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["10.1.0.143"]},"goroutine":7010166,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:22:12.904572527Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"KUBELETUSER","component":"session","clientIP":"10.12.18.51:44164","method":"HEAD","path":"/v2/REPO6/manifests/45decec54fb8db7fd3b1cde8726dd9370fe2e78b","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*"],"Authorization":["******"],"Referer":["https://registry.smen.space/v2/REPO6/manifests/45decec54fb8db7fd3b1cde8726dd9370fe2e78b"],"User-Agent":["containerd/v1.6.34"],"X-Forwarded-For":["10.1.0.148"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["10.1.0.148"]},"goroutine":7010171,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T09:22:13.190392294Z","message":"HTTP API"}

shcherbak · 2025-03-06T18:55:20Z

PATCH 500 and surrounding logs (grep -C3 -A3)

{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.51:28692","method":"PATCH","path":"/v2/REPO1/blobs/uploads/d8fee716-39a0-40ea-b5a8-de5f439bb3f8","statusCode":202,"latency":"3m55s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["529667453"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"]},"goroutine":64789,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:28:24.953249238Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.53:30316","method":"PATCH","path":"/v2/REPO2/blobs/uploads/2dc4bffb-30ed-4675-bb82-0499adce7caf","statusCode":202,"latency":"1m2s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["15777017"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":81827,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:30:11.855047442Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.53:60356","method":"PATCH","path":"/v2/REPO2/blobs/uploads/952e5f6e-7332-428b-a373-b7299ff72d57","statusCode":202,"latency":"1m26s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["9982"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":94682,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:34:09.787062847Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.51:48488","method":"PATCH","path":"/v2/REPO1/blobs/uploads/3f13870c-9c23-400b-acf4-a2fd29976d00","statusCode":500,"latency":"5m1s","bodySize":178,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["531260452"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"]},"goroutine":96520,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:38:13.894181382Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.53:17430","method":"PATCH","path":"/v2/REPO2/blobs/uploads/fa7b6d6a-8d75-4def-b1b7-0070774797b4","statusCode":202,"latency":"1m29s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["15780574"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":113463,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:39:09.721435791Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.53:23650","method":"PATCH","path":"/v2/REPO2/blobs/uploads/84134250-bea7-433f-b18e-2edfe9bcebc4","statusCode":202,"latency":"1m32s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["9982"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":115556,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:39:57.087661955Z","message":"HTTP API"}
{"level":"info","clusterMember":"10.12.18.52:8080","clusterMemberIndex":"1","module":"http","username":"GITLABUSER","component":"session","clientIP":"10.12.18.53:16534","method":"PATCH","path":"/v2/REPO2/blobs/uploads/c8d37a18-a54c-41c7-aa9e-3cee24f1cee1","statusCode":202,"latency":"1m58s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["9989"],"User-Agent":["docker/24.0.5 go/go1.20.6 git-commit/a61e2b4 kernel/5.8.0-0.bpo.2-amd64 os/linux arch/amd64 UpstreamClient(Docker-Client/27.5.0 \\(linux\\))"],"X-Forwarded-For":["REDACTED"],"X-Forwarded-Proto":["https"],"X-Real-Ip":["REDACTED"],"X-Zot-Cluster-Hop-Count":["1"]},"goroutine":132981,"caller":"zotregistry.dev/zot/pkg/api/session.go:137","time":"2025-03-06T12:44:49.161654806Z","message":"HTTP API"}

…t locking - lock per repo on pushes/pulls/retention, in short index operations - lock per digest when using multiple operations affecting the cachedb and storage (blob writes/deletes/moves/links in storage which need to be in accordance with cachedb content) Do not lock multiple repos at the same time in the same goroutine! It will cause deadlocks. Same applies to digests. Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

…ilures Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

…aining image cache Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

andaaron changed the title ~~eat(storage): enable parallel writes by using per-repo locking~~ feat(storage): enable parallel writes by using per-repo locking Feb 15, 2025

andaaron force-pushed the storage2 branch 7 times, most recently from 21c56e6 to a83648b Compare February 22, 2025 13:12

andaaron force-pushed the storage2 branch 4 times, most recently from b1540db to d5b8180 Compare February 23, 2025 08:51

andaaron changed the title ~~feat(storage): enable parallel writes by using per-repo locking~~ feat(storage): enable parallel writes by using per-repo and per-digest locking Feb 23, 2025

andaaron mentioned this pull request Feb 23, 2025

feat(storage): enable parallel writes by using per-repo locking #2600

Closed

andaaron marked this pull request as ready for review February 23, 2025 09:36

andaaron requested review from roxanaN and rchincha as code owners February 23, 2025 09:36

andaaron requested a review from eusebiu-constantin-petu-dbk February 23, 2025 09:37

rchincha mentioned this pull request Feb 27, 2025

[Bug]: Significant performance hit due to excessive image store lock #2964

Open

andaaron force-pushed the storage2 branch 4 times, most recently from 6f48c6d to cd30d2d Compare February 27, 2025 13:38

andaaron force-pushed the storage2 branch from cd30d2d to 77adc1e Compare February 27, 2025 14:17

andaaron requested a review from rchamarthy February 27, 2025 17:55

andaaron force-pushed the storage2 branch from 77adc1e to 8db4c06 Compare February 28, 2025 23:06

andaaron mentioned this pull request Mar 2, 2025

[Bug]: Push stack because of restoring deduped blobs #2999

Open

andaaron added 3 commits March 9, 2025 19:14

chore: fix killing zot and collecting logs in case of cluster test fa…

b81028b

…ilures Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

fix: handle indexes / manifest lists for buildkit manifest lists cont…

e8bf6e5

…aining image cache Signed-off-by: Andrei Aaron <aaaron@luxoft.com>

andaaron force-pushed the storage2 branch from c5d95a0 to e8bf6e5 Compare March 9, 2025 19:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(storage): enable parallel writes by using per-repo and per-digest locking #2968

feat(storage): enable parallel writes by using per-repo and per-digest locking #2968

andaaron commented Feb 15, 2025 •

edited

Loading

codecov bot commented Feb 15, 2025 •

edited

Loading

rchincha commented Feb 27, 2025

andaaron commented Feb 27, 2025

andaaron commented Feb 27, 2025

shcherbak commented Mar 2, 2025

rchincha commented Mar 2, 2025

rchincha commented Mar 3, 2025 •

edited

Loading

andaaron commented Mar 3, 2025

rchincha commented Mar 5, 2025

andaaron commented Mar 5, 2025

rchincha commented Mar 5, 2025

andaaron commented Mar 5, 2025

andaaron commented Mar 6, 2025 •

edited

Loading

shcherbak commented Mar 6, 2025

andaaron commented Mar 6, 2025

shcherbak commented Mar 6, 2025

shcherbak commented Mar 6, 2025 •

edited

Loading

shcherbak commented Mar 6, 2025 •

edited

Loading

feat(storage): enable parallel writes by using per-repo and per-digest locking #2968

Are you sure you want to change the base?

feat(storage): enable parallel writes by using per-repo and per-digest locking #2968

Conversation

andaaron commented Feb 15, 2025 • edited Loading

codecov bot commented Feb 15, 2025 • edited Loading

Codecov Report

rchincha commented Feb 27, 2025

andaaron commented Feb 27, 2025

andaaron commented Feb 27, 2025

shcherbak commented Mar 2, 2025

rchincha commented Mar 2, 2025

rchincha commented Mar 3, 2025 • edited Loading

andaaron commented Mar 3, 2025

rchincha commented Mar 5, 2025

andaaron commented Mar 5, 2025

rchincha commented Mar 5, 2025

andaaron commented Mar 5, 2025

andaaron commented Mar 6, 2025 • edited Loading

shcherbak commented Mar 6, 2025

andaaron commented Mar 6, 2025

shcherbak commented Mar 6, 2025

shcherbak commented Mar 6, 2025 • edited Loading

shcherbak commented Mar 6, 2025 • edited Loading

andaaron commented Feb 15, 2025 •

edited

Loading

codecov bot commented Feb 15, 2025 •

edited

Loading

rchincha commented Mar 3, 2025 •

edited

Loading

andaaron commented Mar 6, 2025 •

edited

Loading

shcherbak commented Mar 6, 2025 •

edited

Loading

shcherbak commented Mar 6, 2025 •

edited

Loading