Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some https://nodejs.org/dist/ builds are experiencing intermittent 500 errors #32683

Closed
ryanolee opened this issue Apr 6, 2020 · 45 comments
Closed

Comments

@ryanolee
Copy link

ryanolee commented Apr 6, 2020

  • Version: Many (see list)
  • Platform: All
  • Subsystem: Dist build system

See:
nodejs/nodejs.org#3075
nodejs/build#2263

For other node repos effected by this.

What steps will reproduce the bug?

Many /dist/ builds are giving 500 errors. Such as https://nodejs.org/dist/v0.11.15/

How often does it reproduce? Is there a required condition?

Temperamental as it currently stands. Currently 500-ing for me

What is the expected behavior?

Dist zip download

What do you see instead?

500 errors

Additional information

A number of other issues have been raised about this:
#32680
#32681
#32682

We will probably want to flatten them into one global issue surrounding builds across all of these versions failing to properly load.

Currently building a list of effected URLS
Thus far failing builds can be found here <- Likley to change given 500 errors seem almost random at this stage

@ryanolee ryanolee changed the title All https://nodejs.org/dist/ builds are experiencing temperamental 500 errors Some https://nodejs.org/dist/ builds are experiencing temperamental 500 errors Apr 6, 2020
@aabuhijleh
Copy link

Facing this issue as well

@vitali-raikov
Copy link

vitali-raikov commented Apr 6, 2020

Seems that it affects all or most builds. If you try downloading the same file multiple times then it's hit or miss, I get approximately 70% internal server error and remaining 30% it downloas dsuccesfully, probably there are some unhealthy instances in nodejs.org load balancer.

@WesleyKlop
Copy link

WesleyKlop commented Apr 6, 2020

This is not only happening for builds but the entire https://nodejs.org site in general is giving random 500 errors

See also:
nodejs/nodejs.org#3075
nodejs/nodejs.org#3076

@devu4
Copy link

devu4 commented Apr 6, 2020

Only the v12.16.1 version is working at the moment. Seems to be an issue on Node site. I've just used the 12.16.1 version for now but its not going to be a solution for everyone.

@ryanolee ryanolee changed the title Some https://nodejs.org/dist/ builds are experiencing temperamental 500 errors Some https://nodejs.org/dist/ builds are experiencing intermittent 500 errors Apr 6, 2020
@bnoordhuis bnoordhuis pinned this issue Apr 6, 2020
@pjammula
Copy link

pjammula commented Apr 6, 2020

Facing same issue. Is there any workaround?

@Raul6469
Copy link

Raul6469 commented Apr 6, 2020

Facing same issue. Is there any workaround?

You can try to use Docker images 👍

daniellockyer added a commit to TryGhost/Admin that referenced this issue Apr 6, 2020
no issue

- using the setup-node action is currently failing because of an
  upstream issue - nodejs/node#32683
- this commit switches to using a Docker container
@Rayanox
Copy link

Rayanox commented Apr 6, 2020

Same issue ..

@mmodi
Copy link

mmodi commented Apr 6, 2020

Same issue. https://nodejs.org/dist/v9.11.1/node-v9.11.1-linux-x64.tar.gz file is missing

@dalawwa
Copy link

dalawwa commented Apr 6, 2020

Only the v12.16.1 version is working at the moment. Seems to be an issue on Node site. I've just used the 12.16.1 version for now but its not going to be a solution for everyone.

I only managed to have the xz version working, gz did not work for 12.16.1.

@steelion
Copy link

steelion commented Apr 6, 2020

We are experiencing the problems for a few hours from now?
Do you have any update on these Issues?
current stable "node-v12.16.1-linux-x64.tar.gz" is not working. ;.(

@ashvin777
Copy link

Same issue... it is intermittent.

@19h
Copy link
Contributor

19h commented Apr 6, 2020

This is blocking our deploy pipeline, how can we help here? It seems rather weird that a project as big as Node has issues reported as far back as 8 hours and there's nothing done about it..

@dominicjeremiah2020
Copy link

Is there an update on this issue? This is blocking our build pipelines and we are closing in on a release deadline.

@aviatrix
Copy link

aviatrix commented Apr 6, 2020

Use apt/yum to install the package for your respective distro.

@AshCripps
Copy link
Member

@nodejs/build-infra

@j-blandford
Copy link

Same Issue. Blocking our release + deployment pipeline totally. Yikes

@max-kuzmin
Copy link

You, guys, should store copies of all third-party packages in local repository to prevent such failures

@aviatrix
Copy link

aviatrix commented Apr 6, 2020

You, guys, should store copies of all third-party packages in local repository to prevent such failures

At a big corp i used to work we did that, it saved our asses few times npm has had issues. it's a good idea if you have the resources :)
edit: not in a repository per se, but a caching layer that acts as the proxy between you and npm.
This also makes downloading packages faster

@mosaadm
Copy link

mosaadm commented Apr 6, 2020

It worked for me just now, maybe already fixed ?

@fishcharlie
Copy link

fishcharlie commented Apr 6, 2020

You, guys, should store copies of all third-party packages in local repository to prevent such failures

@MaxGsomGsom Some prebuilt systems simply aren’t designed for that. In an ideal world that would be the case. But the practicality of it in many cases is unreasonable or untenable.

@johnpangalos
Copy link

I made a ticket for the build team as well nodejs/build#2263

@AshCripps
Copy link
Member

Hi everyone, Node and Build are aware of issues people are seeing. Unfortunatly the downloads server has access limited to only @nodejs/build-infra which means we have to wait untill one of that team is available to fix the issue. Please keep telling us what you see to help diagnose (though please dont open new issues to avoid clutter) Sorry for any inconvenience caused!

@chmielas
Copy link

chmielas commented Apr 6, 2020

@AshCripps e.g.

node-gyp rebuild ERR! stack Error: 500 response downloading https://nodejs.org/download/release/v6.12.3/node-v6.12.3-headers.tar.gz

@MelaFelec
Copy link

Hi, I'm having this issue :
Could not download node-linux-x64.tar.gz (org.nodejs:node:10.14.0)
> Could not get resource 'https://nodejs.org/dist/v10.14.0/node-v10.14.0-linux-x64.tar.gz'.
> Premature end of Content-Length delimited message body (expected: 18622101; received: 16547593

Hope it helps to resolve the problem.

@jgehrcke
Copy link
Contributor

jgehrcke commented Apr 6, 2020

Please keep telling us what you see to help diagnose

We see super slow downloads or aborting downloads. From tonight:

[2020-04-06T01:42:04Z] Downloading https://nodejs.org/dist/v12.16.1/node-v12.16.1-linux-x64.tar.gz...
[2020-04-06T01:44:40Z] curl: (92) HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)

@addaleax
Copy link
Member

addaleax commented Apr 6, 2020

I made a ticket for the build team as well nodejs/build#2263

^

This is the relevant issue where you can expect updates. It’s not necessary to comment here with “this is happening for me too” comments – as Ash said, we’re aware of the problem.

@chingc
Copy link

chingc commented Apr 6, 2020

This issue happens almost once a month and it's honestly very frustrating. :(

kremalicious referenced this issue in kremalicious/gatsby-plugin-matomo Apr 6, 2020
@dbashford
Copy link

Definitely the first time I remember experiencing this in 3 or 4 years of our builds yanking node down. Right now its working 40% of the time. When it doesn't work, it just 500s at the outset. No partial loading.

@Hubert-Hoffman
Copy link

Could not download node-linux-x64.tar.gz (org.nodejs:node:10.15.0)
> Could not get resource 'https://nodejs.org/dist/v10.15.0/node-v10.15.0-linux-x64.tar.gz'.
> Read timed out

@mhdawson
Copy link
Member

mhdawson commented Apr 6, 2020

I'll go to the machine to take a look to see if I can see anything obvious as a starting point.

@mhdawson
Copy link
Member

mhdawson commented Apr 6, 2020

Will use nodejs/build#2263 to comments.

@jgehrcke
Copy link
Contributor

jgehrcke commented Apr 6, 2020

I'll go to the machine

Sorry for asking... the [single] machine serving all nodejs.org downloads in the world? :D

@fishcharlie
Copy link

@jgehrcke

Sorry for asking... the [single] machine serving all nodejs.org downloads in the world? :D

Forget where I read it. But I think Cloudflare handles a lot of caching for Node. So it's very possible it is just a single machine, especially with how aggressively Cloudflare can caching stuff.

@bnb
Copy link
Contributor

bnb commented Apr 6, 2020

Sorry for asking... the [single] machine serving all nodejs.org downloads in the world? :D

I assume he meant his computer.

Since @mhdawson has pointed to nodejs/build#2263 as the single source of truth for this discussion, I'm going to lock this discussion in favor of that one. There are already further updates there ❤️

@nodejs nodejs locked as resolved and limited conversation to collaborators Apr 6, 2020
@bnb bnb closed this as completed Apr 6, 2020
@bnb
Copy link
Contributor

bnb commented Apr 6, 2020

(Going to leave pinned until resolved so we don't see duplicate issues created.)

@sam-github
Copy link
Contributor

@bnb is it possible to leave it open and locked? That might better reflect the situation, if technically possible.

@bnb bnb reopened this Apr 6, 2020
@bnb
Copy link
Contributor

bnb commented Apr 6, 2020

Done ✅

@bnoordhuis bnoordhuis unpinned this issue Apr 7, 2020
@bnoordhuis
Copy link
Member

It's been resolved for some time now, closing.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests