V2 revised #349

hannahhoward · 2022-10-04T22:59:43Z

Goals

Get a scoped down version of v2 revised without the transport refactor that is compatible with graphsync master

Implementation

Essentially, this PR starts from the point in v2 where I began doing transport refactors.

The transport refactor was huge, and speculative from the beginning because we never got a second transport. I did a lot of changes, but never had confidence in the ultimate result, and more importantly, I still haven't succcessfully run the go-fil-markets integration tests with that branch. This is despite multiple attempts at it.

This PR drops the transport refactor part of data transfer v2, but ports over compatibility with graphsync and various quality of life improvements to the data transfer codebase. I have confirmed that it passes integration tests in go-fil-markets.

The path from here to shipping go-data-transfer v2 is:

update go-graphsync to go-libp2p v0.22, tag
update this branch to go-libp2p v.22, merge, tag
update the go-fil-markets integration branch to go-fil-markets master and merge and tag

I think I can get these remaining tasks done quickly if this gets the ok.

willscott · 2022-10-05T01:19:24Z

channels/channels_fsm.go

@@ -160,57 +160,65 @@ var ChannelEvents = fsm.Events{
 			return nil
 		}),

+	// TODO: There are four states from which the request can be "paused": request, queued, awaiting acceptance
+	// and ongoing. There four states of being


'there are 4 states of being paused' ?

willscott · 2022-10-05T01:20:37Z

channels/internal/migrations/migrations.go

+	// RequiresFinalization indicates at the end of the transfer, the channel should
+	// be left open for a final settlement
+	RequiresFinalization bool
+	// Stages traces the execution fo a data transfer.


willscott · 2022-10-05T01:21:06Z

impl/events.go

+	if err != nil {
+		return err
+	}
+	chst, err := m.channels.GetByID(context.TODO(), chid)


shouldn't need .TODO()?

This is a long standing issue but we really don't have a context for the the data transfer manager and we don't use the context passed to OpenDataTransferChannel well. Will fix seperately.

codecov-commenter · 2022-10-07T07:12:57Z

Codecov Report

Merging #349 (aa4d634) into master (5b2a311) will increase coverage by 1.82%.
The diff coverage is 66.32%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #349      +/-   ##
==========================================
+ Coverage   65.97%   67.79%   +1.82%     
==========================================
  Files          29       22       -7     
  Lines        3436     2826     -610     
==========================================
- Hits         2267     1916     -351     
+ Misses        816      694     -122     
+ Partials      353      216     -137

Impacted Files	Coverage Δ
channelmonitor/channelmonitor.go	`75.33% <ø> (ø)`
impl/environment.go	`77.77% <ø> (ø)`
impl/receiver.go	`47.16% <ø> (-30.19%)`	⬇️
network/libp2p_impl.go	`66.47% <ø> (ø)`
tracing/tracing.go	`93.75% <ø> (ø)`
message/message1_1prime/message.go	`50.00% <28.57%> (-4.40%)`	⬇️
message/message1_1prime/transfer_message.go	`23.07% <40.00%> (+3.07%)`	⬆️
message/message1_1prime/transfer_request.go	`63.49% <46.15%> (-5.48%)`	⬇️
impl/events.go	`46.93% <56.52%> (-30.15%)`	⬇️
impl/impl.go	`53.69% <56.52%> (-11.26%)`	⬇️
... and 18 more

hannahhoward · 2022-10-07T21:09:39Z

@rvagg can I get an OK to merge from you?

rvagg · 2022-10-07T22:51:43Z

go.mod

 	github.com/ipfs/go-ipld-format v0.2.0
 	github.com/ipfs/go-log/v2 v2.5.1
 	github.com/ipfs/go-merkledag v0.5.1
 	github.com/ipfs/go-unixfs v0.3.1
-	github.com/ipld/go-ipld-prime v0.16.0
+	github.com/ipld/go-ipld-prime v0.17.1-0.20220624062450-534ccf82237d


we have an 0.18 we should be using now

rvagg

yeah, ok, a big dose of 🤞 from me but it's time to get this all merged and exercised.
the fsm stuff around the new paused states are a bit fuzzy for me, there's quite a bit of extra complexity being introduced when those states flow through the system

given the expected breaking changes, it's time to setup the v2 release BREAKING CHANGE: v2 modules

* feat(revalidators): initial refactor refactor revalidation process -- removing independent revalidations, making validations results more clear * refactor(rename): RevalidateToComplete -> RequiresFinalization * refactor(datatransfer): enhance validator comments * refactor(message): revert message response changes * refactor(impl): comment and refactor on events add comments to event processing and also extract functions for receiving requests to a new file * refactor(message): s/IsVoucherResult/IsValidationResult rename the IsVoucherResult to is 'IsValidationResult' also add a method for generating messages from validation results * feat(events): only dispatch events on change Only dispatch Pause, Resume, SetDataLimit, and RequiresFinalization on change * style(imports): fix imports * feat(events): add DataLimitExceeded event * Update channels/channel_state.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * Update manager.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * Update manager.go Co-authored-by: dirkmc <dirkmdev@gmail.com> * Update statuses.go Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: dirkmc <dirkmdev@gmail.com> Co-authored-by: Rod Vagg <rod@vagg.org>

* refactor(validators): remove revalidation move to all revalidation being asynchronous * feat(validators): implied pauses causes datalimit exceeded and requires finalization to leave request paused, regardless of where LeaveRequestPaused is set * refactor(events): reorder events reorder validation events so all get record when transfer finishes * Update impl/impl.go Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Rod Vagg <rod@vagg.org>

* feat(ipld): vouchers as plain ipld.Node * feat: add ValidationResult#Equals() utility * feat(ipld): introduce TypedVoucher tuple type * chore(ipld): ipld.Node -> datamodel.Node * chore: remove RegisterVoucherResultType * fix: minor staticcheck fixes

use cborgencomptaiblenode to simply channelstate interface

* chore(deps): update libp2p v0.19.4 (#341) * feat(ipld): use bindnode/registry Co-authored-by: Hannah Howard <hannah@hannahhoward.net>

Ports state machine and graphsync changes without the big transport refactor

also upgrades graphsync and removes go-libp2p-core paths

hannahhoward marked this pull request as ready for review October 4, 2022 23:00

hannahhoward requested review from rvagg and willscott October 4, 2022 23:03

willscott approved these changes Oct 5, 2022

View reviewed changes

rvagg reviewed Oct 7, 2022

View reviewed changes

rvagg approved these changes Oct 7, 2022

View reviewed changes

hannahhoward and others added 11 commits October 7, 2022 16:31

chore(v2): setup the v2 release

c9f768b

given the expected breaking changes, it's time to setup the v2 release BREAKING CHANGE: v2 modules

chore(message): delete old message format code (#330)

1b3f897

refactor(channelstate): use cborgencompatiblenode (#336)

f8d2cd0

use cborgencomptaiblenode to simply channelstate interface

feat(ipld): use bindnode/registry (#340)

cb04997

* chore(deps): update libp2p v0.19.4 (#341) * feat(ipld): use bindnode/registry Co-authored-by: Hannah Howard <hannah@hannahhoward.net>

refactor(v2): port graphsync w/o transport refactor

491841f

Ports state machine and graphsync changes without the big transport refactor

fix(pr): respond to pr comments

7d75380

chore(deps): upgrade libp2p v0.22

5638aad

also upgrades graphsync and removes go-libp2p-core paths

fix(deps): use tagged go-ipld-prime

aa4d634

hannahhoward force-pushed the v2-revised branch from ffaafed to aa4d634 Compare October 7, 2022 23:34

hannahhoward merged commit c9889dc into master Oct 8, 2022

rvagg deleted the v2-revised branch January 5, 2023 01:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

V2 revised #349

V2 revised #349

hannahhoward commented Oct 4, 2022 •

edited

Loading

willscott Oct 5, 2022

willscott Oct 5, 2022

willscott Oct 5, 2022

hannahhoward Oct 7, 2022

codecov-commenter commented Oct 7, 2022 •

edited

Loading

hannahhoward commented Oct 7, 2022

rvagg Oct 7, 2022

rvagg left a comment

V2 revised #349

V2 revised #349

Conversation

hannahhoward commented Oct 4, 2022 • edited Loading

Goals

Implementation

willscott Oct 5, 2022

Choose a reason for hiding this comment

willscott Oct 5, 2022

Choose a reason for hiding this comment

willscott Oct 5, 2022

Choose a reason for hiding this comment

hannahhoward Oct 7, 2022

Choose a reason for hiding this comment

codecov-commenter commented Oct 7, 2022 • edited Loading

Codecov Report

hannahhoward commented Oct 7, 2022

rvagg Oct 7, 2022

Choose a reason for hiding this comment

rvagg left a comment

Choose a reason for hiding this comment

hannahhoward commented Oct 4, 2022 •

edited

Loading

codecov-commenter commented Oct 7, 2022 •

edited

Loading