Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flaky Test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion #1276

Closed
shahzadlone opened this issue Apr 2, 2023 · 4 comments
Closed

Flaky Test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion #1276

shahzadlone opened this issue Apr 2, 2023 · 4 comments
Labels
area/p2p Related to the p2p networking system area/testing Related to any test or testing suite bug Something isn't working

Comments

@shahzadlone
Copy link
Member

Flakey test - failed to validate if peer is a DHT peer

=== FAIL: tests/integration/net/state/simple/replicator TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion (unknown)
2023-04-02T00:33:36.447-0400, INFO, defra.tests.integration, , {"Database": "badger-in-memory"}
2023-04-02T00:33:36.453-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:36.455-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:36.456-0400    ERROR   peerstore/ds    pstoreds/keybook.go:63  error when fetching pubkey from datastore for peer 12D3KooWCfaJ9UdJtFrnDtvWPXHd4kNad74w2Hh1xRSodLdkTPtr: datastore closed

2023-04-02T00:33:36.456-0400    ERROR   peerstore/ds    pstoreds/keybook.go:81  error while updating pubkey in datastore for peer 12D3KooWCfaJ9UdJtFrnDtvWPXHd4kNad74w2Hh1xRSodLdkTPtr: datastore closed

2023-04-02T00:33:36.459-0400    ERROR   peerstore/ds    pstoreds/keybook.go:63  error when fetching pubkey from datastore for peer 12D3KooWEnPHF1TyNCDsrdM6Mf9VKg3S4w3tLoS6s77frweWzvLK: datastore closed

2023-04-02T00:33:36.459-0400    ERROR   peerstore/ds    pstoreds/keybook.go:81  error while updating pubkey in datastore for peer 12D3KooWEnPHF1TyNCDsrdM6Mf9VKg3S4w3tLoS6s77frweWzvLK: datastore closed

2023-04-02T00:33:36.460-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:36.460-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:36.519-0400, INFO, defra.tests.integration, Starting P2P node, {"P2P address": "/ip4/0.0.0.0/tcp/0"}
2023-04-02T00:33:36.669-0400, INFO, defra.node, Created LibP2P host, {"PeerId": "12D3KooWJz77LD5somZ4gYvJNfLRqrkDhrzcphHvcAKkgTd2zSBa", "Address": ["/ip4/0.0.0.0/tcp/0"]}
2023-04-02T00:33:36.679-0400, INFO, defra.net, Starting internal broadcaster for pubsub network
2023-04-02T00:33:36.745-0400, INFO, defra.tests.integration, Starting P2P node, {"P2P address": "/ip4/0.0.0.0/tcp/0"}
2023-04-02T00:33:36.843-0400, INFO, defra.node, Created LibP2P host, {"PeerId": "12D3KooWCxEVqHBWfftaWTgdUhkPNWm4LGtrih6kb9cMyfDHL8ai", "Address": ["/ip4/0.0.0.0/tcp/0"]}
2023-04-02T00:33:36.855-0400, INFO, defra.net, Starting internal broadcaster for pubsub network
2023-04-02T00:33:36.945-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWCxEVqHBWfftaWTgdUhkPNWm4LGtrih6kb9cMyfDHL8ai", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "JOINED"}
2023-04-02T00:33:36.946-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWJz77LD5somZ4gYvJNfLRqrkDhrzcphHvcAKkgTd2zSBa", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "JOINED"}
2023-04-02T00:33:36.992-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiflruukcvmnz234tmrf4b74o24xgojc3ngptz3ygt7f3ljyreceze", "CID": "bafybeiexbrccs4xe3sgegdoptbzygir26lkz2p7nnfjkxojlzxtgvgn7eu", "Height": 2}
2023-04-02T00:33:37.000-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiarm3we4ayugdhnopebxu74kupxjlplggoxeqfhi2a4bgyqtmktl4", "CID": "bafybeictbh7k6v4mimgtvo6xujxleie7zzlecgkaiakckhyjk4sp3k5yva", "Height": 2}
2023-04-02T00:33:37.011-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiarm3we4ayugdhnopebxu74kupxjlplggoxeqfhi2a4bgyqtmktl4", "CID": "bafybeictbh7k6v4mimgtvo6xujxleie7zzlecgkaiakckhyjk4sp3k5yva", "Height": 2}
2023-04-02T00:33:37.030-0400, ERROR, defra.net, Error processing log, {"DocKey": {"CollectionID":"","InstanceType":"","DocKey":"bae-43deba43-f2bc-59f4-9056-fef661b22832","FieldId":""}, "CID": "bafybeifetnvrnokdswejeyyvxufio4jsxclfbolfxg3j7dgks2hiygimr4", "Error": "Couldn't find field Email for doc {  bae-43deba43-f2bc-59f4-9056-fef661b22832 }"}
2023-04-02T00:33:37.037-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiflruukcvmnz234tmrf4b74o24xgojc3ngptz3ygt7f3ljyreceze", "CID": "bafybeiexbrccs4xe3sgegdoptbzygir26lkz2p7nnfjkxojlzxtgvgn7eu", "Height": 2}
2023-04-02T00:33:37.042-0400, INFO, defra.tests.integration, , {"RequestResults": [{"Email":"imnotyourbuddyguy@source.ca","Name":"Shahzad"}]}
2023-04-02T00:33:37.044-0400, INFO, defra.tests.integration, , {"RequestResults": [{"Name":"Shahzad"}]}
2023-04-02T00:33:37.053-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWJz77LD5somZ4gYvJNfLRqrkDhrzcphHvcAKkgTd2zSBa", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "LEFT"}
2023-04-02T00:33:37.059-0400, INFO, defra.db, Closing DefraDB process...
2023-04-02T00:33:37.118-0400, INFO, defra.db, Successfully closed running process
2023-04-02T00:33:37.119-0400    ERROR   peerstore/ds    pstoreds/keybook.go:63  error when fetching pubkey from datastore for peer 12D3KooWJz77LD5somZ4gYvJNfLRqrkDhrzcphHvcAKkgTd2zSBa: datastore closed

2023-04-02T00:33:37.128-0400, INFO, defra.db, Closing DefraDB process...
2023-04-02T00:33:37.130-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.130-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.131-0400    ERROR   peerstore/ds    pstoreds/keybook.go:63  error when fetching pubkey from datastore for peer 12D3KooWCxEVqHBWfftaWTgdUhkPNWm4LGtrih6kb9cMyfDHL8ai: datastore closed

2023-04-02T00:33:37.131-0400    ERROR   peerstore/ds    pstoreds/keybook.go:81  error while updating pubkey in datastore for peer 12D3KooWCxEVqHBWfftaWTgdUhkPNWm4LGtrih6kb9cMyfDHL8ai: datastore closed

2023-04-02T00:33:37.178-0400, INFO, defra.db, Successfully closed running process
2023-04-02T00:33:37.179-0400, INFO, defra.tests.integration, , {"Database": "defra-memory-datastore"}
2023-04-02T00:33:37.182-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.182-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.183-0400, INFO, defra.tests.integration, Starting P2P node, {"P2P address": "/ip4/0.0.0.0/tcp/0"}
2023-04-02T00:33:37.233-0400, INFO, defra.node, Created LibP2P host, {"PeerId": "12D3KooWL2uGxw3f5da9HqzoXeFq9dqb2en3NpbNhfR9kWtGrm3y", "Address": ["/ip4/0.0.0.0/tcp/0"]}
2023-04-02T00:33:37.242-0400, INFO, defra.net, Starting internal broadcaster for pubsub network
2023-04-02T00:33:37.244-0400, INFO, defra.tests.integration, Starting P2P node, {"P2P address": "/ip4/0.0.0.0/tcp/0"}
2023-04-02T00:33:37.282-0400, INFO, defra.node, Created LibP2P host, {"PeerId": "12D3KooWSGUvLTMvZabyqBLuKZmb2auA2zXwUxt9UZ3S5mL8vwgf", "Address": ["/ip4/0.0.0.0/tcp/0"]}
2023-04-02T00:33:37.286-0400, INFO, defra.net, Starting internal broadcaster for pubsub network
2023-04-02T00:33:37.364-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWSGUvLTMvZabyqBLuKZmb2auA2zXwUxt9UZ3S5mL8vwgf", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "JOINED"}
2023-04-02T00:33:37.365-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWL2uGxw3f5da9HqzoXeFq9dqb2en3NpbNhfR9kWtGrm3y", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "JOINED"}
2023-04-02T00:33:37.383-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiflruukcvmnz234tmrf4b74o24xgojc3ngptz3ygt7f3ljyreceze", "CID": "bafybeiexbrccs4xe3sgegdoptbzygir26lkz2p7nnfjkxojlzxtgvgn7eu", "Height": 2}
2023-04-02T00:33:37.386-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiarm3we4ayugdhnopebxu74kupxjlplggoxeqfhi2a4bgyqtmktl4", "CID": "bafybeictbh7k6v4mimgtvo6xujxleie7zzlecgkaiakckhyjk4sp3k5yva", "Height": 2}
2023-04-02T00:33:37.401-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiarm3we4ayugdhnopebxu74kupxjlplggoxeqfhi2a4bgyqtmktl4", "CID": "bafybeictbh7k6v4mimgtvo6xujxleie7zzlecgkaiakckhyjk4sp3k5yva", "Height": 2}
2023-04-02T00:33:37.413-0400, ERROR, defra.net, Error processing log, {"DocKey": {"CollectionID":"","InstanceType":"","DocKey":"bae-43deba43-f2bc-59f4-9056-fef661b22832","FieldId":""}, "CID": "bafybeifetnvrnokdswejeyyvxufio4jsxclfbolfxg3j7dgks2hiygimr4", "Error": "Couldn't find field Email for doc {  bae-43deba43-f2bc-59f4-9056-fef661b22832 }"}
2023-04-02T00:33:37.421-0400, INFO, defra.merkleclock, Replacing DAG head, {"Old": "bafybeiflruukcvmnz234tmrf4b74o24xgojc3ngptz3ygt7f3ljyreceze", "CID": "bafybeiexbrccs4xe3sgegdoptbzygir26lkz2p7nnfjkxojlzxtgvgn7eu", "Height": 2}
2023-04-02T00:33:37.424-0400, INFO, defra.tests.integration, , {"RequestResults": [{"Email":"imnotyourbuddyguy@source.ca","Name":"Shahzad"}]}
2023-04-02T00:33:37.428-0400, INFO, defra.tests.integration, , {"RequestResults": [{"Name":"Shahzad"}]}
2023-04-02T00:33:37.429-0400, INFO, defra.net, Received new pubsub event, {"SenderId": "12D3KooWL2uGxw3f5da9HqzoXeFq9dqb2en3NpbNhfR9kWtGrm3y", "Topic": "bae-43deba43-f2bc-59f4-9056-fef661b22832", "Message": "LEFT"}
2023-04-02T00:33:37.436-0400, INFO, defra.db, Closing DefraDB process...
2023-04-02T00:33:37.436-0400, INFO, defra.db, Successfully closed running process
2023-04-02T00:33:37.440-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.440-0400    ERROR   dht     go-libp2p-kad-dht@v0.21.1/subscriber_notifee.go:108     could not check peerstore for protocol support: err: datastore closed
2023-04-02T00:33:37.442-0400, INFO, defra.db, Closing DefraDB process...
2023-04-02T00:33:37.444-0400    ERROR   peerstore/ds    pstoreds/keybook.go:63  error when fetching pubkey from datastore for peer 12D3KooWSGUvLTMvZabyqBLuKZmb2auA2zXwUxt9UZ3S5mL8vwgf: datastore closed

2023-04-02T00:33:37.444-0400    ERROR   peerstore/ds    pstoreds/keybook.go:81  error while updating pubkey in datastore for peer 12D3KooWSGUvLTMvZabyqBLuKZmb2auA2zXwUxt9UZ3S5mL8vwgf: datastore closed
@shahzadlone shahzadlone added bug Something isn't working area/testing Related to any test or testing suite area/p2p Related to the p2p networking system labels Apr 2, 2023
@shahzadlone shahzadlone mentioned this issue Apr 2, 2023
Open
@shahzadlone shahzadlone changed the title Flakey test - failed to validate if peer is a DHT peer Flakey test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion Apr 2, 2023
@shahzadlone shahzadlone changed the title Flakey test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion Flakey Test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion Apr 2, 2023
@jsimnz
Copy link
Member

jsimnz commented Apr 2, 2023

This doesn't look like the other flakey tests.

@shahzadlone
Copy link
Member Author

I have seen this a few times now on my machine. The underlying cause might be different than the other flakey tests.

@jsimnz jsimnz changed the title Flakey Test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion Flaky Test - TestP2PReplicatorUpdateWithNewFieldSyncsDocsToOlderSchemaVersion Apr 6, 2023
@shahzadlone
Copy link
Member Author

Might be a similar one encountered here: https://github.com/sourcenetwork/defradb/actions/runs/4649721058/jobs/8228265743?pr=1304

@shahzadlone
Copy link
Member Author

Closing seems resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/p2p Related to the p2p networking system area/testing Related to any test or testing suite bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jsimnz @shahzadlone