feat: load static websites and paths

[SgtPooki]

• chore: debugging fetch speed
• fix(libp2p-patch): filter out bad peers
• feat: load static websites

demo

Pre-reqs

You have to visit the preview site first, and make sure the SW is loaded. Once it is, the below links should work for you.

Note: You may need to refresh a few times.. i'm sure there is additional pre-fetching we need to do so we can ensure sites load properly.

Links

ipfs.tech/images directory listing page: https://bafybeigrqbyusjs5hh3e7upxhwm52hqocygezkvke3c6neobrhon4yt3v4.on.fleek.co/helia-sw/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images

ipfs.tech/images/images/ipfs-desktop-hex.png: https://bafybeigrqbyusjs5hh3e7upxhwm52hqocygezkvke3c6neobrhon4yt3v4.on.fleek.co/helia-sw/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images/ipfs-desktop-hex.png

ipfs.tech website: https://bafybeigrqbyusjs5hh3e7upxhwm52hqocygezkvke3c6neobrhon4yt3v4.on.fleek.co/helia-sw/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ

You can also test the loading of content from the main page, and link to the direct content by clicking the call-to-action button beneath

[SgtPooki]

Some issues brought up via slack:

1. If I click on ipfs-applications-diagram.png I see a directory listing for index.html and ipfs-applications-diagram.png. Clicking either entry give me a blank page with Not found
   • URL is then https://bafybeihzimkzei2dye7omtm6banteyl4kavaut4bfdvyfvaq4pefnf5hlu.on.fleek.co/helia-sw/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images/images/ipfs-applications-diagram.png/index.html

That .png file is actually a directory

╰─ ✔ ❯ ipfs ls QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images
Qma5hgVwaYzQGXCvf3BXB5Y7J86Z8fDWQz5kYgV1hbVNCs 17878  command-line-hex.png
Qmeosk9pDjpmMZGbuRpPjaWdNnjMdTpxTthbB8seYfRAcB -      ipfs-applications-diagram.png/
QmQfoXavfiGLmUL4bRnvS6w8LAwRw4nWLSL9jF7qU1e2B3 9904   ipfs-cluster.png
QmXkkYJYye8tzqL19Fyt1oea73yPWp3LH6774nmF7i9tie 7839   ipfs-companion-hex.png
QmVmBnuqByeEqoLu1HkCn2zYeMZ2L3ooPDKyWV5pfaiydJ 6268   ipfs-desktop-hex.png
QmU58Dt54RbV6ccJNLWtrPmzxMy6u2ndpeBACGsKWFPdYa 20434  js-ipfs-logo.png
QmPGLQcMexuZvZ2jeX4UhWofD4JeVucet5Uw58tvAN7tsn 16255  kubo-hex-logo.png
QmeGaLaMx1bH1z57KS1qLkoTCLPAQZbDsnAavjhupkaUqx 909480 meet-builders-thumbnail-pinata.png
Qmcm32sVsMYhURY3gqH7vSQ76492t5Rfxb3vsWCb35gVme 231643 social-card.png

╰─ ✔ ❯ ipfs ls QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images/ipfs-applications-diagram.png
QmaNPC2vHmMBTWtkhHmEGAojWMusidq63jFKvNczaSBim5 394    index.html
QmdmV2xUjRhhS2vAbtUPs837rKunfXZn8zHmTKu59bR649 937808 ipfs-applications-diagram.png

The not-found at site/helia-sw/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ/images/images/ipfs-applications-diagram.png/index.html is definitely a bug, i will look into it.

2. The CID of the UnixFS directory being traversed needs to be in the subdomain somehow - otherwise if I have two websites I’m rendering:
   https://bafyfoo.on.fleek.co/helia-sw/QmGood
   https://bafyfoo.on.fleek.co/helia-sw/QmEvil
   QmEvil can read any cookies set by QmGood which leads to all sorts of credential-stealing vulnerabilities

I don't think we can do subdomains without getting a server/dns involved somehow. Each subdomain/origin would be a different service-worker instance, but we could re-use the same service worker code.

[SgtPooki]

Others via slack:

Is putting the path into the URL bar viable as compared to using URL fragments with some escaping for existing fragments? I've noticed that if I refresh a page I end up with an error (probably from Fleek trying to resolve it) like ipfs resolve -r /ipfs/bafybeigqtacsovumsjmo7xgiaydqlp4jaqtwpuvciyhfasb4q3jeus3fzq/ipfs/QmeUdoMyahuQUPHS2odrZEL6yk2HnNfBJ147BeLXsZuqLJ: no link named "ipfs" under bafybeigqtacsovumsjmo7xgiaydqlp4jaqtwpuvciyhfasb4q3jeus3fzq

Since I changed the intercept path to be helia-sw we shouldn't run into the resolve -r /ipfs/ blah errors from fleek anymore. maybe i'm missing something.

Do we need this approach with paths to deal with range-requests for fragments of data like video?

Range requests are currently not implemented.. I was going to tackle that next. I also need to make sure ETag and caching is working properly, because it doesn't seem to be working 100% right now

[SgtPooki]

@achingbrain fixed dir listing, thanks for calling that out. See video

helia-sw-dir-list-mostly-fixed.mp4

[whizzzkid]

That looks like one hell of an example.

Q:

• My SW never connects to peers, what could be the issue?
• This work needs cleanup.
• More comments from examples POV.
• How would this even be tested?
• Concerns in comments.

[whizzzkid]

does this even work? since this is a new worker, it won't have access to the helia instance from the old install.

You might not need this, but if you still, a better way to do this would be in the registration.

[SgtPooki]

yea, i need an onUnregister for the sw, but I may not need to, haven't dove into it fully, but i'm also wondering if we actually want a persistent helia instance, or if we want to boot one up on every request.

I imagine that keeping the same instance running, with a timeout to stop it when no work is done in X period, might be the best solution for now.

[whizzzkid]

this sounds super dangerous, wondering is an injection is feasible.

[SgtPooki]

more than likely with this current code, yes.

[SgtPooki]

That looks like one hell of an example.

Q:

• My SW never connects to peers, what could be the issue?

Sometimes I have to use bootstrapOnly and add my own peerID, but ipni should be working properly for you, may need a live debugging session so i can observe directly.

• This work needs cleanup.

yes 100%, it's pretty messy, but i'm just trying to get things to a working point and can polish easily later. PR is more for testing than for code review.

A lot of code here exists just to debug why things don't work when they aren't working. (like the setInterval)

• How would this even be tested?

as in unit/e2e tests?

e2e: probably with local node and a bootStrapOnly config to that specific node

unit: should be fairly straightforward for most of the code.