Taskcluster "tools/ unittests (Python 3)" failing on all PRs

[foolip]

Example: https://tools.taskcluster.net/groups/Xy9azuGlT1Oke4HBMbGS_g

This is blocking everything :/

[foolip]

Once the cause is fixed, we'll also have to retrigger all PRs affected by this.

[gsnedders]

+ sudo apt install -qqy python3-pip
The following additional packages will be installed:
  dh-python libpython3-dev libpython3.6 libpython3.6-dev python3-asn1crypto
  python3-cffi-backend python3-crypto python3-cryptography python3-dev
  python3-distutils python3-idna python3-keyring python3-keyrings.alt
  python3-lib2to3 python3-secretstorage python3-setuptools python3-wheel
  python3-xdg python3.6-dev
Suggested packages:
  python-crypto-doc python-cryptography-doc python3-cryptography-vectors
  gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0
  python-secretstorage-doc python-setuptools-doc
The following NEW packages will be installed:
  dh-python libpython3-dev libpython3.6 libpython3.6-dev python3-asn1crypto
  python3-cffi-backend python3-crypto python3-cryptography python3-dev
  python3-distutils python3-idna python3-keyring python3-keyrings.alt
  python3-lib2to3 python3-pip python3-secretstorage python3-setuptools
  python3-wheel python3-xdg python3.6-dev
0 upgraded, 20 newly installed, 0 to remove and 2 not upgraded.
Need to get 48.1 MB of archives.
After this operation, 90.4 MB of additional disk space will be used.
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3-stdlib-extensions/python3-lib2to3_3.6.7-1~18.04_all.deb  404  Not Found [IP: 91.189.91.23 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3-stdlib-extensions/python3-distutils_3.6.7-1~18.04_all.deb  404  Not Found [IP: 91.189.91.23 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3.6/libpython3.6_3.6.7-1~18.04_amd64.deb  404  Not Found [IP: 91.189.91.23 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3.6/libpython3.6-dev_3.6.7-1~18.04_amd64.deb  404  Not Found [IP: 91.189.91.23 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/p/python3.6/python3.6-dev_3.6.7-1~18.04_amd64.deb  404  Not Found [IP: 91.189.91.23 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[gsnedders]

Basically, we're using an out of date apt database (presumably only updating when we update the image), and we're running into the eviction of old packages from the archive.ubuntu.com pool.

That said, given the installs are in .taskcluster.yml, we run into the fact that TC only supports using the config in pull/xxx/head (https://bugzilla.mozilla.org/show_bug.cgi?id=1539523), and therefore affected PRs need rebased.

[foolip]

Oh, we need to rebase, opening and closing won't help? Can you comment on the PRs where the author has to rebase?

[gsnedders]

@foolip Yes. If the issue is in .taskcluster.yml, it needs a rebase. :/

[foolip]

Sad face reaction emoji.