Large file support, 504 Gateway time outs

[bmann]

Summary

We rely on the IPFS protocol, and an ipfs node running on your local machine, to upload / sync files with our servers. If you are on a slow connection or are uploading large files (at this time, mainly videos), fission up can fail because of time outs syncing the files.

fission up on a directory will fail with a not very useful "Something went wrong" error message. Running DEBUG shows a 504 error, which is an IPFS gateway time out. Running fission up multiple times will sometimes end up in success, at which point DNS will get updated, as the file(s) make their way through the IPFS network and are successfully backed up in our servers.

Problem

Bad developer experience and confusing error messages when publishing using fission up with large files.

Impact

People get frustrated and think our service is broken, and give up using it.

We can't reliably support large files.

Solution

For large files, implement a notification:

All files added to IPFS locally. N files over X detected, which will background sync.

Might need something similar for large numbers of files? Not sure:

Over 500 files detected, background sync is in progress.

This also means our CLI will need to run in the background and either poll or get "push" notifications in some way from our service.

In practice, sending a 504 Gateway time out is not ideal. That is being passed through from our hosted IPFS node directly. Don't know if we can intercept that at some other layer, because it's pretty directly in the protocol.

[bmann]

I'm currently fission up'ing a node_modules directory. Let's skip for the moment whether or not this is the correct thing to do :)

That's lots of small files, and same issue -- no feedback on the CLI, I assume it's just recursing through all those tiny folders and adding them and it will take a while to actually complete.

In the meantime, the CLI looks like it's hanging, with no user feedback -- including no clock emoji. I'll let it run for a while, and then I'll run it again with DEBUG to see if I can see anything interesting other than pages of stuff scrolling by.

[expede]

Related to #338

[expede]

@bmann is this still an issue?

[bmann]

@expede I don't think we can consider this done without doing #312 or similar.

e.g. even the way we walk an entire directory to do an entire IPFS add, when lots of the files are already pinned, could likely be optimized in some way.

Let's close this particular issue and wait for user feedback / thinkabout creative ways to tackle this.