starting some docs for our next meeting discussions #15
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
might be deployed across hardware
valeriupredoi
requested changes
Oct 3, 2022
| If however, it is possble to break down the logic of `_from_storage` so that individual "part_chunks" which are contiguous are processed on the storage side of the LAN (in the ost or nearby) then meaningful performance is possible. The Python client in the end would simply see a list of partial products of the computational method which have come direct from the osts. It will not care whether those parts came from a different breakdown of storage than it anticpated in the chunking (though of course the storage will need to do the necessary mapping to generate the partial sums). | ||
|
|
||
| In the longer term, where we expect that we will have to pass a decompression method down through the `_decode_chunk` interface, it _will_ be necessary for the computational storage to respect the `_decode_chunk` interface server-side. This is of course what is required with S3, where we effectively need an S3 proxy to do the work. It might be that this is also required in some implementations of POSIX storage systems if they wish to implement computational storage OR they will need to somehow respect block contiguous placement in some way. | ||
|
|
There was a problem hiding this comment.
@bnlawrence I think this is probably better suited in the README?
There was a problem hiding this comment.
Sure, but for the moment this is intended for discussion this Friday ...
There was a problem hiding this comment.
ah okay then it's fine to have it in the pseudo-docs repo, cheers @bnlawrence
|
cheers for this @bnlawrence - I think any operational example should probably go in the README for now, before we actually get sphinx-ed dox, feel free to say no to that - I also requested changes to get you used with a GH review that requests changes (in case you're not used to, am not sure of your GH user level, apols if I grossly underestimated it 😁 ) |
|
(Not possible to underestimate my GH foo) |
valeriupredoi
approved these changes
Oct 4, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These docs may or may not yet render as proper diagrams