Uploading artifacts to the registry and downloading them

[aguschin]

Citing an article from @francesco086 https://medium.com/@francesco.calcavecchia/we-refused-to-use-a-hammer-on-a-screw-story-of-a-gto-based-model-registry-c540ac5d129f:

(relates to #83)

---

• We think that one should be able to operate on an artifact registry using only gto commands. dvc put and dvc get should be “overloaded” by gto.
• As we have seen, there is an issue with the dvc remote secrets. We are not fully satisfied with the existing solutions. Is there a better way?

We suggest the interface for two new gto commands:

gto put [OPTIONS] NAME TARGETS [TARGETS ...]
  Put an artifact in a registry
Positional Arguments:
  NAME     Artifact name  [required]
  TARGETS  Input files/directories to add [required]
Options:
  --repo          Git repository, local (path) or remote (url) [default: .]
  -R              Recursively add files under directory targets
  --glob          Allows targets containing shell-style wildcards
  --dvc           Use dvc to track and upload to remote targets
  --bump-major    Bump major version
  --bump-minor    Bump minor version
  --bump-patch    Bump patch version

gto pick [OPTIONS] NAME VERSION
Pick an artifact from a registry
Positional Arguments:
  NAME     Artifact name  [required]
  VERSION  Artifact version [required]
Options:
  --repo          Git repository, local (path) or remote (url) [default: .]

[francesco086]

Hi @aguschin, I think I can finally find a little bit of spare time to work on this :)

I was thinking, what about starting from the put functionality, restricted to the simple upload of a file. Then pick.

On a second iteration we could then add the --dvc option to support large files...?

What do you think?

[aguschin]

@francesco086, had a discussion about this and #337 with @dberenbaum. Can you help me remember what was the reason to have put and pick? Was it because you had repo1 with experiments and repo2 with models only? Was that separation made to have more granular access control? Or there was another reason? Thanks!

[francesco086]

Hi @aguschin ! Yes, in a way. But the real reason, as mentioned in #337, it is because otherwise gto as artifact registry is incomplete.