[6.4.0/1/2] `GLIBC_2.29' not found - not running on Rocky Linux 8 #30612

TimmiORG · 2023-10-10T06:12:03Z

Description:

RocketChat 6.4.0 and 6.4.1 are not working on my rocky linux 8 VM.

Steps to reproduce:

Updated from 6.3.9 to 6.4.0 / 6.4.1 or 6.4.2

Expected behavior:

RocketChat able to start.

Actual behavior:

RocketChat does not start.

Server Setup Information:

Version of Rocket.Chat Server: 6.4.1
Operating System: Rocky Linux 8.8
Deployment Method: tar
Number of Running Instances: 1
DB Replicaset Oplog: yes
NodeJS Version: 14.21.3
MongoDB Version: 5.0.21

Client Setup Information

Desktop App or Browser Version: Desktop App 3.9.0
Operating System: Fedora 38

Additional context

The GLIBC version of Rocky Linux 8.8 is 2.28.

Relevant logs:

Oct 10 08:05:24 rocket-test RocketChat[2166]: /opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:218
Oct 10 08:05:24 rocket-test RocketChat[2166]:      throw error;
Oct 10 08:05:24 rocket-test RocketChat[2166]:      ^
Oct 10 08:05:24 rocket-test RocketChat[2166]: Error: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /opt/Rocket.Chat/programs/server/npm/node_modules/isolated-vm/out/isolated_vm.node)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Object.Module._extensions..node (internal/modules/cjs/loader.js:1173:18)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.load (internal/modules/cjs/loader.js:979:32)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.Mp.load (/opt/Rocket.Chat/programs/server/runtime.js:37:33)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Function.Module._load (internal/modules/cjs/loader.js:819:12)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (internal/modules/cjs/loader.js:1003:19)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at require (internal/modules/cjs/helpers.js:107:18)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Object.<anonymous> (/opt/Rocket.Chat/programs/server/npm/node_modules/isolated-vm/isolated-vm.js:1:-44)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module._compile (internal/modules/cjs/loader.js:1114:14)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.Mp._compile (/opt/Rocket.Chat/programs/server/runtime.js:77:23)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.load (internal/modules/cjs/loader.js:979:32)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.Mp.load (/opt/Rocket.Chat/programs/server/runtime.js:37:33)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Function.Module._load (internal/modules/cjs/loader.js:819:12)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (internal/modules/cjs/loader.js:1003:19)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at require (internal/modules/cjs/helpers.js:107:18)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at npmRequire (/opt/Rocket.Chat/programs/server/npm-require.js:111:12)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.useNode (packages/modules-runtime.js:751:18)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at module (/opt/Rocket.Chat/programs/server/packages/modules.js:204170:8)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at fileEvaluate (packages/modules-runtime.js:336:7)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (packages/modules-runtime.js:238:14)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at module (app/integrations/server/lib/isolated-vm/isolated-vm.ts:2:43)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at fileEvaluate (packages/modules-runtime.js:336:7)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (packages/modules-runtime.js:238:14)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at module (app/integrations/server/lib/triggerHandler.js:1:1149)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at fileEvaluate (packages/modules-runtime.js:336:7)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (packages/modules-runtime.js:238:14)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at server/services/meteor/service.ts:1:72
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at module (server/services/meteor/service.ts:285:2)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at fileEvaluate (packages/modules-runtime.js:336:7)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.require (packages/modules-runtime.js:238:14)
Oct 10 08:05:24 rocket-test RocketChat[2166]:    at Module.moduleLink [as link] (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/modules/node_modules/@meteorjs/reify/lib/runtime/index.js:52:22) {
Oct 10 08:05:24 rocket-test RocketChat[2166]:  code: 'ERR_DLOPEN_FAILED'
Oct 10 08:05:24 rocket-test RocketChat[2166]: }

The text was updated successfully, but these errors were encountered:

dazoot · 2023-10-10T07:15:39Z

I have the same issue on Debian 10.13.

knordsiek · 2023-10-10T09:43:05Z

Yes thats correct.
You have to upgrade to Rocky Linux 9.
https://linuxiac.com/upgrade-rocky-linux-8-to-rocky-linux-9/
Hint: you have to check the correct Version in the repository before starting with upgrade

Debian 10.13 you have to upgrade to Debian 11.

TimmiORG · 2023-10-10T09:54:16Z

@knordsiek
This is the official statement from RocketChat?
Rocky Linux 9 is the minumum for Rocket.Chat 6.4.x releases?

knordsiek · 2023-10-10T10:00:28Z

No its only my personal experience.
Background: libm.so.6 needs glibc 2.29
Rocky Linux 8 is using glibc 2.28
It is very difficult to change a running system to a newer glibc it's the better way to upgrade the system

TimmiORG · 2023-10-10T10:03:42Z

OK,
I can't see any information that Rocky Linux 9 will be required. So I guess this is not intended to be required.

knordsiek · 2023-10-10T10:16:07Z

Upgrade to glibc 2.29 possible on Rocky Linux 8?
See this forum thread to answer your question:

https://forums.rockylinux.org/t/upgrade-glibc-2-28-to-2-29/7558

TimmiORG · 2023-10-10T10:18:02Z

As I wrote before. This should not be correct.
Waiting for offical answer.

TimmiORG · 2023-10-11T07:40:03Z

Hi @sampaiodiego are you able to comment on this one?

Gummikavalier · 2023-10-11T09:11:33Z

RC 6.4.1 works for us and we are running RHEL8 that has GNU C Library (GNU libc) stable release version 2.28. according to command /lib64/libc.so.6.

So it may be a trickier problem. CentOS7 obviously was already becoming obsolete with RC (segfaulting in node binary), and upgrades from that have been pretty much mandatory if you run tar install of RC.

(Or maybe RedHat has some additional backporting, but if the requirement is the version number 2.29, it should not work for us either ATM.)

TimmiORG · 2023-10-11T09:28:23Z

RC 6.4.1 works for us and we are running RHEL8 that has GNU C Library (GNU libc) stable release version 2.28. according to command /lib64/libc.so.6.

So it may be a trickier problem. CentOS7 obviously was already becoming obsolete with RC (segfaulting in node binary), and upgrades from that have been pretty much mandatory if you run tar install of RC.

(Or maybe RedHat has some additional backporting, but if the requirement is the version number 2.29, it should not work for us either ATM.)

@Gummikavalier
are you running docker or tar version?

Gummikavalier · 2023-10-11T10:56:06Z

@Gummikavalier are you running docker or tar version?

Tar

TimmiORG · 2023-10-11T11:13:04Z

@Gummikavalier
hm....
Did you perform any other steps between 6.3.x and 6.4.x?

Gummikavalier · 2023-10-11T16:13:21Z

No special steps. We actually updated from 6.3.10 to 6.4.1 in our production just yesterday, but in the test environment from older 6.3.x to 6.4.0 immediately when it was released. Neither had issues.

Nodejs version has been in the 14.21.3 for a long time now, and we run MongoDB 5.0.21 too.

For the record our rocket.chat-6.4.1.tgz package sha256sums is:

$ sha256sum rocket.chat-6.4.1.tgz 
e2d6f4cc03517cd4ca4e9f8e6e4319b6e999ec38355ee382504ba1df0078fdd6  rocket.chat-6.4.1.tgz

Gummikavalier · 2023-10-12T06:07:09Z

There was a glibc update to cover buffer overflow vulnerability this month. But the new version didn't seem to play into the issue either, at least not in our setup.
https://seclists.org/oss-sec/2023/q4/18

TimmiORG · 2023-10-12T06:53:57Z

@Gummikavalier thank you for your messages.
Really strange that this is working for you. I will do some more tests later today.

Gummikavalier · 2023-10-12T08:08:20Z

Yup. Also a note: we have several RC setups, all installed at different times within 7 years so their history is also bit different, and they all work ok with the glibc 2.28 of RHEL 8.8.

TimmiORG · 2023-10-12T11:42:03Z

I just installed a new Rocky Linux 8 and I run into the same issue. So really no idea what is causing it.
Using official NodeJS14 and MongoDB 5.

Gummikavalier · 2023-10-12T13:36:52Z

Odd, we are too. I'm stumped with this. 😕

Our NPM version is

$ npm -v
6.14.18

if that matters. It used to but there hasn't been mention of it lately on the releases page.

The last version it was mentioned was NPM: 6.14.17:
https://github.com/RocketChat/Rocket.Chat/releases/tag/6.2.5

TimmiORG · 2023-10-12T14:08:49Z

Using the same npm version.

Gummikavalier · 2023-10-13T09:39:02Z

Do you get this or similar 403 Forbidden errors during the building of RC?
#29600

We do get these errors. If you don't that could be the difference, and your instance would load something that is incompatible with glibc 2.28, while ours cannot.

TimmiORG · 2023-10-16T05:39:07Z

Hi @Gummikavalier
no I don't get such messages.
This is how it looks like.

[root@rocket-test ~]# cd /tmp/bundle/programs/server && npm install
npm WARN deprecated node-pre-gyp@0.15.0: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future

> fibers@5.0.1 install /tmp/bundle/programs/server/node_modules/fibers
> node build.js || nodejs build.js

`linux-x64-83-glibc` exists; testing
Binary is fine; exiting
npm WARN lifecycle meteor-dev-bundle@~install: cannot run in wd meteor-dev-bundle@ node npm-rebuild.js (wd=/tmp/bundle/programs/server)
added 158 packages from 84 contributors and audited 158 packages in 6.355s

5 packages are looking for funding
  run `npm fund` for details

found 2 moderate severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details


   ╭────────────────────────────────────────────────────────────────╮
   │                                                                │
   │      New major version of npm available! 6.14.18 → 10.2.0      │
   │   Changelog: https://github.com/npm/cli/releases/tag/v10.2.0   │
   │               Run npm install -g npm to update!                │
   │                                                                │
   ╰────────────────────────────────────────────────────────────────╯

Gummikavalier · 2023-10-16T06:08:34Z

Ok. That indeed confirms some things but no more about that. Thank you! :)

But yes, then our install indeed differs from yours, and it may be possible that we could replicate your issue by doing certain things. (Too tricky to arrange for me.)

You could try disabling outbound access on your server during the build time so that RC would have to rely only on the modules that come with the RC tar bundle.

Edit: And run npm cache clean first to clear up the existing already downloaded stuff.

TimmiORG · 2023-10-16T06:56:31Z

@Gummikavalier
Just tried but it does not make any difference.

Gummikavalier · 2023-10-16T17:09:02Z

Thanks. I've run out of ideas on this apart from that maybe RockyLinux8 is then not based on RHEL8 sources comparably enough. 😞

rouralberto · 2023-10-17T03:06:41Z

This is happening to me using the official Docker Image upgrading from 6.3.10 to latest 6.4.1.

KarolKski · 2023-10-17T05:31:06Z

Having the same issue using the official 6.4.1 docker Image

bitfridge · 2023-10-17T07:58:13Z

Same for me. Got a reminder to update, pulled the latest release with tag "6" and ran into this issue. Version 6.3.10 (tag "6.3") works for me. Running with docker 24.0.6 on Debian 12.

alimodev · 2023-10-18T15:34:01Z

Same problem using the 6.4.1 official image.

Error: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found (required by /app/bundle/programs/server/npm/node_modules/isolated-vm/out/isolated_vm.node)

ldd --version shows version 2.28

derjohn · 2023-10-19T09:29:42Z

Same problem using the 6.4.2 official docker image, but it works with 6.4.2-alpine docker image.
FYI @alimodev

sampaiodiego · 2023-10-19T11:44:10Z

thx everyone for your comments.. I'll take a look what is happening to our official Docker image.

sampaiodiego · 2023-10-19T12:38:02Z

docker-library/official-images#15583

TimmiORG · 2023-10-19T12:39:27Z

Hi @sampaiodiego
please note that this is also an issue for the TAR installations.

sampaiodiego · 2023-10-19T12:50:30Z

right @TimmiORG .. I was not aware of such dependency.. it is related to one of the security improvements we've made recently..

it seems the easier approach is to update the system to one that has GLIBC 2.29+ by default.. as previously mentioned it might get tricky to update only glibc..

I'll talk to our internal team in order to make this new requirement something more clear..

Gummikavalier · 2023-10-19T12:53:12Z

@sampaiodiego Hi! Do you have any idea/guess why it works with RHEL8.8 that also has GLIBC2.28?

debdutdeb · 2023-10-19T12:59:21Z

Hi @Gummikavalier no I don't get such messages. This is how it looks like.

[root@rocket-test ~]# cd /tmp/bundle/programs/server && npm install
npm WARN deprecated node-pre-gyp@0.15.0: Please upgrade to @mapbox/node-pre-gyp: the non-scoped node-pre-gyp package is deprecated and only the @mapbox scoped package will recieve updates in the future

> fibers@5.0.1 install /tmp/bundle/programs/server/node_modules/fibers
> node build.js || nodejs build.js

`linux-x64-83-glibc` exists; testing
Binary is fine; exiting
npm WARN lifecycle meteor-dev-bundle@~install: cannot run in wd meteor-dev-bundle@ node npm-rebuild.js (wd=/tmp/bundle/programs/server)
added 158 packages from 84 contributors and audited 158 packages in 6.355s

5 packages are looking for funding
  run `npm fund` for details

found 2 moderate severity vulnerabilities
  run `npm audit fix` to fix them, or `npm audit` for details


   ╭────────────────────────────────────────────────────────────────╮
   │                                                                │
   │      New major version of npm available! 6.14.18 → 10.2.0      │
   │   Changelog: https://github.com/npm/cli/releases/tag/v10.2.0   │
   │               Run npm install -g npm to update!                │
   │                                                                │
   ╰────────────────────────────────────────────────────────────────╯

Don't use root user when running npm install, or add --unsafe-perm there

debdutdeb · 2023-10-19T13:00:35Z

From what @Gummikavalier said, my guess is at install, for him, isolated-vm (or ig prebuild? idk) detected the glibc mismatch and rebuilt the binaries. Which wouldn't happen if you run npm install as root and without --unsafe-perm argument.

debdutdeb · 2023-10-19T13:03:20Z

@Gummikavalier could you run ldd on the isolated-vm binary and share the output?

Gummikavalier · 2023-10-19T13:17:34Z

@debdutdeb

Thanks! Yes, we actually don't build RC tar as root but use a separate account for it.

If this is the correct binary
bundle/programs/server/npm/node_modules/isolated-vm/out/isolated_vm.node

it says:

$ ldd isolated_vm.node
	linux-vdso.so.1 (0x00007ffeb87da000)
	libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fc5c79fe000)
	libm.so.6 => /lib64/libm.so.6 (0x00007fc5c767c000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fc5c7464000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc5c7244000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fc5c6e7f000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc5c808b000)

debdutdeb · 2023-10-19T13:24:22Z

Thanks ! One more command please nm -gD /lib64/libm.so.6 | grep GLIBC_2

Gummikavalier · 2023-10-19T13:33:11Z

@debdutdeb

$ nm -gD /lib64/libm.so.6 | grep GLIBC_2
0000000000000000 A GLIBC_2.15
0000000000000000 A GLIBC_2.18
0000000000000000 A GLIBC_2.23
0000000000000000 A GLIBC_2.24
0000000000000000 A GLIBC_2.2.5
0000000000000000 A GLIBC_2.25
0000000000000000 A GLIBC_2.26
0000000000000000 A GLIBC_2.27
0000000000000000 A GLIBC_2.28
0000000000000000 A GLIBC_2.4

RHEL version:

$ cat /etc/redhat-release 
Red Hat Enterprise Linux release 8.8 (Ootpa)

To clarify; on RHEL8.8 RC works but the OP reported RockyLinux failing. But I noticed it was run as root here by TimmiORG:
#30612 (comment)

debdutdeb · 2023-10-19T13:35:49Z

Thanks man :) yeah this makes sense - as in your environment is nothing magical. It's not defining 2.29 for any reason. Which brings me to running npm install as non-root should fix it right up.

TimmiORG · 2023-10-19T13:41:33Z

@debdutdeb
Cool, can confirm that this is working while not using root to build it.

debdutdeb · 2023-10-19T13:45:00Z

Thanks for confirming :)

debdutdeb · 2023-10-19T13:45:40Z

Closing this then. Feel free to reopen if necessary (or continue conversing here).

TimmiORG changed the title ~~[6.4.0 / 6.4.1] `GLIBC_2.29' not found - not running on Rocky 8~~ [6.4.0 / 6.4.1] `GLIBC_2.29' not found - not running on Rocky Linux 8 Oct 10, 2023

eXtrem0us mentioned this issue Oct 14, 2023

Custom CSS won't overwrite existing values #30621

Closed

TimmiORG changed the title ~~[6.4.0 / 6.4.1] `GLIBC_2.29' not found - not running on Rocky Linux 8~~ [6.4.0/1/2] `GLIBC_2.29' not found - not running on Rocky Linux 8 Oct 17, 2023

debdutdeb closed this as completed Oct 19, 2023

debdutdeb mentioned this issue Oct 19, 2023

Problem with Amazon linux 2 images laverdet/isolated-vm#392

Open

Gummikavalier mentioned this issue Dec 16, 2023

GLIBCXX_3.4.29' not found #31258

Closed

davidjgraph mentioned this issue Feb 25, 2025

26.0.16 AppImage: /lib64/libm.so.6: version `GLIBC_2.29' not found (on Rocky Linux 8.10) jgraph/drawio-desktop#2004

Closed

2 tasks

[6.4.0/1/2] `GLIBC_2.29' not found - not running on Rocky Linux 8 #30612

[6.4.0/1/2] `GLIBC_2.29' not found - not running on Rocky Linux 8 #30612

Comments

TimmiORG commented Oct 10, 2023 • edited Loading

Description:

Steps to reproduce:

Expected behavior:

Actual behavior:

Server Setup Information:

Client Setup Information

Additional context

Relevant logs:

dazoot commented Oct 10, 2023

knordsiek commented Oct 10, 2023

TimmiORG commented Oct 10, 2023

knordsiek commented Oct 10, 2023

TimmiORG commented Oct 10, 2023

knordsiek commented Oct 10, 2023

TimmiORG commented Oct 10, 2023

TimmiORG commented Oct 11, 2023

Gummikavalier commented Oct 11, 2023 • edited Loading

TimmiORG commented Oct 11, 2023

Gummikavalier commented Oct 11, 2023

TimmiORG commented Oct 11, 2023

Gummikavalier commented Oct 11, 2023

Gummikavalier commented Oct 12, 2023 • edited Loading

TimmiORG commented Oct 12, 2023

Gummikavalier commented Oct 12, 2023

TimmiORG commented Oct 12, 2023

Gummikavalier commented Oct 12, 2023 • edited Loading

TimmiORG commented Oct 12, 2023

Gummikavalier commented Oct 13, 2023

TimmiORG commented Oct 16, 2023

Gummikavalier commented Oct 16, 2023 • edited Loading

TimmiORG commented Oct 16, 2023

Gummikavalier commented Oct 16, 2023

rouralberto commented Oct 17, 2023 • edited Loading

KarolKski commented Oct 17, 2023

bitfridge commented Oct 17, 2023

alimodev commented Oct 18, 2023

derjohn commented Oct 19, 2023

sampaiodiego commented Oct 19, 2023

sampaiodiego commented Oct 19, 2023

TimmiORG commented Oct 19, 2023

sampaiodiego commented Oct 19, 2023

Gummikavalier commented Oct 19, 2023

debdutdeb commented Oct 19, 2023

debdutdeb commented Oct 19, 2023 • edited Loading

debdutdeb commented Oct 19, 2023

Gummikavalier commented Oct 19, 2023

debdutdeb commented Oct 19, 2023

Gummikavalier commented Oct 19, 2023 • edited Loading

debdutdeb commented Oct 19, 2023

TimmiORG commented Oct 19, 2023

debdutdeb commented Oct 19, 2023

debdutdeb commented Oct 19, 2023

TimmiORG commented Oct 10, 2023 •

edited

Loading

Gummikavalier commented Oct 11, 2023 •

edited

Loading

Gummikavalier commented Oct 12, 2023 •

edited

Loading

Gummikavalier commented Oct 12, 2023 •

edited

Loading

Gummikavalier commented Oct 16, 2023 •

edited

Loading

rouralberto commented Oct 17, 2023 •

edited

Loading

debdutdeb commented Oct 19, 2023 •

edited

Loading

Gummikavalier commented Oct 19, 2023 •

edited

Loading