Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Emit an io.Reader for streaming blobs #11

Closed
wants to merge 1 commit into from

Conversation

JordonPhillips
Copy link
Member

@JordonPhillips JordonPhillips commented Mar 31, 2020

This updates blobs with the streaming trait to io.Reader instead of []byte. They'll now look like:

import (
	"io"
)

type MyStruct struct {
	StreamingBlob io.Reader
}

Or if they have the mediatype trait too:

import (
	"io"
)

type CityImageData io.Reader

func (m CityImageData) MediaType() string {
	return "image/jpeg"
}

Note that this PR is dependent on this Smithy change. This PR is still merge-able. It'll run, but the code path is effectively dead until the Smithy PR is merged and this generator updates to that version.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Copy link
Member

@skmcgrail skmcgrail left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note, since io.Reader is an interface we don't need to generate it as a *io.Reader instead it can just be io.Reader

Copy link
Contributor

@jasdel jasdel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this account for the behavior differences between an stream used as input vs a stream used as output? For output stream, the member type should be an io.ReadCloser, Where as input should only be io.Reader.

This is under the assumption that streams must only be on the operation's top level input shapes. Does smithy allow a non-toplevel input/output shape to contain a Stream member ?

@jasdel
Copy link
Contributor

jasdel commented Apr 1, 2020

To extend on @skmcgrail comment, generated interface types should never be used as pointers. they should always be value as a general rule. Since interfaces are already a pointer, similar to maps and slices.

@JordonPhillips
Copy link
Member Author

Does smithy allow a non-toplevel input/output shape to contain a Stream member?

Nope, wouldn't really make any sense.

Will this account for the behavior differences between an stream used as input vs a stream used as output?

No, you can't guarantee a given shape isn't used for only output or only input. I know we discussed making custom types for operation inputs / outputs, so that might be a customization we can do then.

@jasdel
Copy link
Contributor

jasdel commented Apr 2, 2020

type CityImageData io.Reader

Smithy won't be able to create a type alias of an interface. Go does not allow type alias of interfaces.

The MediaType shapes probably will need to structures that contain a underlying reader. What populatees that media type's reader will be based on how it is used. e.g. streaming vs nested.

type CityImageData struct {
     io.Reader
}

type AsCityImageData(r io.Reader) *CityImageData { return &CityImageData{Reader: r} }

Downside of this approach is that if the underlying io.Reader provided by the user has any additional optimizations like ReadFrom etc, those will not be used by utilities like io.Copy because Go type assertions will not query embedded interface implementations. Potentially could generate these helpers onto the generated MediaType to compensate, as a fallback.

@jasdel
Copy link
Contributor

jasdel commented Apr 2, 2020

We'll also want to be aware that the the existing SDK's designs treat input and output streams differently. Input stream being be an io.Reader, and output io.ReadCloser.

The smithy generation could consider requiring an io.ReadCoser for input as well, but this is uncommon outside of the http.Client's Request since most ownership of Closing is expected to be the caller, not the callee.

Utilities like ioutil.NopCloser are available if a user doesn't want Close to be called or dont' want to use a type that implements io.Closer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants