File system content not always available on Fission gateway

[icidasset]

When setting the data root, we'll want to wait on the API to tell us the content was successfully fetched from our local js-ipfs node. In other words, the data from our file system lives on our local js-ipfs node and we need to wait until that is transferred to the Fission gateway.

More notes:

• The user cannot close the window before this action is finished, show a dialogue when trying to close the window.
• If this were not solved, you can have an infinite loader on your app and possibly lose your file system

[expede]

@icidasset I believe that this should fix it: fission-codes/fission#396

I believe that you’ll get a 200 after the data the data fetch is complete — it’s not fire-and-forget internally.Please let me know if you still have problems

[icidasset]

I don't think it's fully working as intended.

Here's how I'm testing this:

• I open https://quotes.fission.app/
• Wait for DNSLink-update message to appear in the console
• Double check if the update-data-root request has succeeded
• Open https://quotes.fission.app/ in another browser
• Check if the CID is the same
• Wait for it to load file system
• 🛑 Infinite spinner

Notes:
If open the same file system in Drive in the same browser, it does load a bit quicker than in other browsers (where it often doesn't load at all).

[expede]

If open the same file system in Drive in the same browser, it does load a bit quicker than in other browsers (where it often doesn't load at all).

This sounds like it’s client side. WebRTC or the WSS connection, maybe?

EDIT: Oh, I think I misread. In the same browser. Gotcha 👍 I think we need to do a much more detailed root cause analysis here. If you have that window open in your browser, there should always be a route to the data since it’s live

[icidasset]

If you have that window open in your browser, there should always be a route to the data since it’s live

Yeah exactly, that's what I don't understand.
The js-ipfs instance where the data lives is still live, so why does the other js-ipfs instance not find it? 🤔

[expede]

Just double checked the server code — we wait for a response from the gateway, so you shouldn’t see a response from us until it’s working. I’m going to double check that we don’t like... ignore errors or something deeper in the code, but on the surface it looks pretty much okay.

[expede]

Yeah, we pass back any errors from the gateway

[expede]

If you have that window open in your browser, there should always be a route to the data since it’s live

Yeah exactly, that's what I don't understand.
The js-ipfs instance where the data lives is still live, so why does the other js-ipfs instance not find it? 🤔

We need to test browser-to-browser content fetching. Validate that it works without the server in between. Maybe add some text locally, and see if you can fetch it on another tab via our SDK from the console. If that works, so the same with a different browser, then a different machine.

Really validate our core assumptions

[icidasset]

Ok, a lot of things to look at here.

Discovery

We need to test browser-to-browser content fetching

There is no browser-to-browser content fetching at the moment, I think? How would the two nodes know about each other without the server in between?

I guess ideally IPFS would automatically discover nodes in the local network, but I guess that's not the case yet for js-ipfs? go-ipfs does have a LAN DHT, pretty sure, but I don't know about js-ipfs.

If they did know about each other, the node address would show up in swarm.peers right? But I'm only seeing the peer id from our gateway.

From https://github.com/ipfs/js-ipfs/tree/master/examples/circuit-relaying:

browser nodes can't connect to each other directly because they lack any sort of socket functionality and relaying on specialized rendezvous nodes introduces an undesirable centralization point to the network. A circuit-relay is a way to solve this problem

I guess this kind of answers my question.
I'll enable this circuit relay thing in js-ipfs (marked as experimental)
EDIT: This a node/gateway setting which was already configured, so no help there.

There's also this auto relay thing that's on the todo list for libp2p libp2p/js-libp2p#699 not sure if that's important or not.

EDIT: Added signaling server to the swarm, which seems to help a lot. Now we wait for libp2p to remove the need for signaling servers 🙏

Gateway 504

I need to observe the response from the update-data-root call more.
I just got the 504 gateway-timeout in the browser (I've had this a bunch of times on the CLI as well).

The API is definitely connected, because the gateway address is listed in ipfs.swarm.peers()
Is the gateway just overwhelmed with requests/downloads? 🤔

Problem seems to be consistent, with the same CID and different CIDs (in other words, with or without file-system changes). The problem does occasionally go away when refreshing the page.

DNS Cache

Found out that the browser was caching the DNS lookups 😓
Downside of having auto-generated service workers.
Will fix this.
(This is specific to the Quotes app)

EDIT: I think this is fixed, hard to tell with service workers.

[icidasset]

Ok, I think I pretty much solved the discovery issue by adding our signaling server to the swarm list in the config 🎉 Will add this to the default config.