Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Voucher refactor integration #707

Conversation

hannahhoward
Copy link
Collaborator

@hannahhoward hannahhoward commented May 3, 2022

Goals

This PR rewrites the go-fil-markets retrieval provider based on the new data transfer voucher refactor (filecoin-project/go-data-transfer#308, filecoin-project/go-data-transfer#322) as well as the new go-statemachine multi-event application PR (filecoin-project/go-statemachine#37).

Implementation

The retrieval client code is unchanged for now -- this change is intended to be backward compatible for the time being -- even though we could seriously drop a ton of code if we stopped having the client wait for the provider to request payment.

All of the revalidation code is -> gone (hence the big delete/add ratio)

The validation code is mostly the same, but we set data limits + pauses + require finalization

Restarts are now handled seperately, and we handle them by simply looking at the current state of the transfer and setting validation status appropriately.

When a data limit or finalization occurs, we trigger the payment states, which in turn trigger sending an updated validation status + a voucher result that requests payment.

When a new voucher arrives, we trigger the payment states again, which processes the last payment and updates the funds received as needed, then sets validation status based on that.

The long and short of all this is we no longer do any tracking of transfer state directly in the provider -- only funds received, the unique state to this layer.

I'm also dropping all the v0 protocols, which haven't actually been in use since pre-mainnet.

@hannahhoward hannahhoward requested review from dirkmc and rvagg May 3, 2022 21:02
lowerBound = target
target += intervalSize
intervalSize += p.PaymentIntervalIncrease
}
return lowerBound
}

func (p Params) NextInterval(currentInterval uint64) uint64 {
// OutstandingBalance produces the amount owed based on the deal params
// for the given transfer state and funds received
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"given transfer state" - so I see that this method returns the oustanding unsealed price as the "outstanding balance" but if the unsealing price is covered then the "outstanding balance" is something different. So this is really an "OutstandingBalanceForState", is that correct? (no need to rename, you say this in the comment, I'm just making sure I'm reading this right).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. amount transferred will always be zero (or should always be zero) if the unseal price hasn't been paid, as unsealing won't have even begun, which means that transfer will not have begun. :)

I'll do some commenting here.

@hannahhoward hannahhoward force-pushed the feat/voucher-refactor-integration branch from e3548c4 to bab8805 Compare May 12, 2022 00:49
@hannahhoward hannahhoward marked this pull request as ready for review May 12, 2022 00:49
@hannahhoward hannahhoward force-pushed the feat/voucher-refactor-integration branch from ef29a9a to a7d62b8 Compare May 12, 2022 01:10
@hannahhoward hannahhoward mentioned this pull request May 12, 2022
@rvagg
Copy link
Member

rvagg commented May 12, 2022

@hannahhoward

this change is intended to be backward compatible

One minor concern is that ProviderDealState is changing shape: TotalSent, CurrentInterval and LegacyProtocol are all gone now and they're not optional fields. I think this struct is only used for saving state and that cbor-gen should gracefully ignore these (both in and out) and I'm not entirely clear on how the statemachine saving stuff works so it'd be good to hear your confirmation that you think this is OK.

@hannahhoward hannahhoward changed the base branch from master to feat/datatransfer-v2-integration June 2, 2022 23:57
@hannahhoward hannahhoward merged commit 8afdf95 into feat/datatransfer-v2-integration Jun 3, 2022
hannahhoward added a commit that referenced this pull request Oct 12, 2022
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Oct 12, 2022
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Oct 12, 2022
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Oct 14, 2022
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Nov 17, 2022
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Jan 7, 2023
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps
hannahhoward added a commit that referenced this pull request Feb 11, 2023
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps

Update retrievalmarket/types.go

Update storagemarket/types.go

chore(deps): update to rc candidate

chore(dtutils): update for new data transfer configuration interface
hannahhoward added a commit that referenced this pull request Feb 11, 2023
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps

Update retrievalmarket/types.go

Update storagemarket/types.go

chore(deps): update to rc candidate

chore(dtutils): update for new data transfer configuration interface
hannahhoward added a commit that referenced this pull request Feb 18, 2023
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps

Update retrievalmarket/types.go

Update storagemarket/types.go

chore(deps): update to rc candidate

chore(dtutils): update for new data transfer configuration interface

chore(deps): update to rc4
hannahhoward added a commit that referenced this pull request Feb 18, 2023
Voucher refactor integration (#707)

* refactor(storagemarket): update storagemarket

* refactor(retrievalmarket): use new voucher system

refactor for predictability and correctness using new voucher system

* chore(deps): update go-data-transfer v2

* style(lint): prep for pr

* docs(retrievalmarket): add comments

* chore(deps): update go-statemachine

* style(imports): fix imports

chore(retrievalmarket): remove old types (#712)

feat(ipld): bindnode support for all voucher types (#713)

* feat(ipld): new data-transfer ipld vouchers + bindnode

* feat(ipld): simplify ipldutils API

* feat(ipld): use new bindnode registry in go-ipld-prime

Ref: ipld/go-ipld-prime#437

feat(deps): update data transfer 61f0756c

feat(deps): update data transfer and other deps

update to master data transfer with libp2p v0.22.0 plus associated other deps

Update retrievalmarket/types.go

Update storagemarket/types.go

chore(deps): update to rc candidate

chore(dtutils): update for new data transfer configuration interface

chore(deps): update to rc4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants