Node 22 on armvc6 and armv7 get into an out of memory infinite loop

[LaurentGoderre]

Version

22

Platform

Linux 431b32e3d8c4 6.6.31-linuxkit #1 SMP Thu May 23 08:36:57 UTC 2024 armv7l Linux

Subsystem

No response

What steps will reproduce the bug?

docker run --cap-add=SYS_PTRACE -e QEMU_STRACE=1 --rm -it --platform linux/arm/v6 node:22-alpine 

How often does it reproduce? Is there a required condition?

Every time

What is the expected behavior? Why is that the expected behavior?

Container starts

What do you see instead?

Without trace, the container just hangs, with tracing the following log is output forever

mremap(1081462784,4096,8192,0,0,1082133536) = -1 errno=12 (Out of memory)

Additional information

No response

[rotu]

@isheludko, could you take a look? I think this is related to CodeRange allocation logic which seems in your wheelhouse.

---

I tried this on my computer and I don't think it's actually running out of memory like the above message suggests at first glance. It starts counting down and then eventually the memory pointer loops around to negative then underflows to positive again, looping forever:

1 mremap(1082126336,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
1 mremap(1082122240,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
...
1 mremap(4096,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
1 mremap(0,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
1 mremap(-4096,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
...
1 mremap(-2147483648,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)
1 mremap(2147479552,4096,8192,0,0,1082132616) = -1 errno=12 (Out of memory)

BUT on AMD64, I also see the same error many times. But it eventually stops when mremap gives a "Bad address" error instead.

1 mremap(46912504504320,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
1 mremap(46912504500224,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
...
1 mremap(46912496119808,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
1 mremap(46912496115712,4096,8192,0,0,0) = -1 errno=14 (Bad address)
1 open("/proc/self/maps",O_RDONLY|O_LARGEFILE) = 17
1 mmap(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00002aaaaed8c000

[isheludko]

mremap parameters don't seem to be CodeRange related but I'd be interested in investigating this issue.

Unfortunately, I'm not allowed to run docker on my machine. Is there another way of reproducing the issue on x64 Linux? Maybe using qemu?

BTW, IIUC a standalone V8 build (d8 binary) should suffer from the same issue...

Do you mind filing a bug in Chromium bug tracker with instructions not involving docker?

[rotu]

I figured it was CodeRange related because CodeRangeAddressHint::GetAddressHint calls mremap in a loop.

I'm not sure how I'd produce this without docker. Can you use podman?

Here's the full output of: docker run --platform=linux/arm --rm -e QEMU_STRACE=1 -it alpine:edge ash -c 'apk add nodejs-current && node -p 42' > out.txt

out.txt

[isheludko]

Ok, I reproduced the issue with podman. Now I need to learn how to debug this and how to build my binary for alpine linux. I managed to build and run a hello world app but debugging doesn't work for me:

podman run --rm -it alpine:edge ash -c 'apk add vim clang gdb && vim /tmp/foo.cc && clang /tmp/foo.cc -o /tmp/foo -g && /tmp/foo && gdb --args /tmp/foo'

where /tmp/foo.cc is:

#include "stdio.h"

int main() {
  printf("Hello World!\n");
  return 0;
}

All I'm getting from GDB is this:

Reading symbols from /tmp/foo...
(gdb) r
Starting program: /tmp/foo
warning: Error disabling address space randomization: Function not implemented
warning: Could not trace the inferior process.
warning: ptrace: Function not implemented
During startup program exited with code 127.
(gdb)

BTW, I still don't see how CodeRangeAddressHint::GetAddressHint calls mremap at all. The first syscall I see there is open("/proc/self/maps",..) which happens after all the mremap calls are done.

[rotu]

Ok, I reproduced the issue with podman. Now I need to learn how to debug this and how to build my binary for alpine linux. I managed to build and run a hello world app but debugging doesn't work for me

Okay. I'm getting the same, and a little googling tells me gdb doesn't work under qemu (though I don't really know how to use either tool well).

BTW, I still don't see how CodeRangeAddressHint::GetAddressHint calls mremap at all. The first syscall I see there is open("/proc/self/maps",..) which happens after all the mremap calls are done.

Good call. It's not unlikely I misread the code and accidentally thought code in OS::RemapShared was in OS::GetFreeMemoryRangesWithin.

It very well could be in d8.cc inside MultiMappedAllocator::AllocateUninitialized. I wrote this code off because (1) I thought this code is not used in Node, which has its own debugger and (2) it has a comment that "The purpose is to allow stability-testing...":
https://github.com/v8/v8/blob/7ab385cda6fcbfc09907bf9a1f5b96da5ec40ffa/src/d8/d8.cc#L321-L342

I still get the same problem with docker run --platform=linux/arm --rm -it alpine:edge ash -c 'apk add nodejs-current && node --no-multi-mapped-mock-allocator -p 42' (though maybe that flag kicks in too late or doesn't do what I think it does).

[isheludko]

MultiMappedAllocator (and the whole d8.cc source) is included only in d8 standalone binary used for running .js files for V8 testing purposes. Thus this code must not be included into NodeJS binary.

I'll keep on trying to move forward with this issue but with a lower priority.

[snowdream]

MultiMappedAllocator (and the whole d8.cc source) is included only in d8 standalone binary used for running .js files for V8 testing purposes. Thus this code must not be included into NodeJS binary.

I'll keep on trying to move forward with this issue but with a lower priority.

It is important, I can not package node 19+ into alpine(docker), I will get into an out of memory infinite loop.(armvc6 and armv7,s390x,etc )

[rotu]

Noodled around with this some more.
I can reproduce this behavior on both a Mac M3 and a Windows x64 computer but not a Raspberry Pi 3B with
Linux hostname 6.6.74+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.74-1+rpt1 (2025-01-27) aarch64

I have found additionally that:

1. Using docker run with memory limits does not help.
2. On my Mac, running this under a QEMU-simulated raspi 3B with a 32-bit kernel and with the docker daemon running inside does not trigger this issue.
3. Running node under setarch does not ameliorate the issue e.g. setarch --32bit --addr-compat-layout node -p 1

[rotu]

This doesn't seem to loop forever on my Windows x64 PC. It does perform lots of mremaps such that time node -p 42 takes 7.5 seconds on qemu-arm versus 1.0 seconds on qemu-aarch64.

The end of the equivalent loop on arm looks like:

1 mremap(73728,4096,8192,0,0,1082132600) = -1 errno=12 (Out of memory)
1 mremap(69632,4096,8192,0,0,1082132600) = -1 errno=12 (Out of memory)
1 mremap(65536,4096,8192,0,0,1082132600) = -1 errno=12 (Out of memory)
1 mremap(61440,4096,8192,0,0,1082132600) = -1 errno=14 (Bad address)
1 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x43dc9000
1 mmap2(NULL,8192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x43dca000
1 mmap2(NULL,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x43dcc000
1 mmap2(0x59200000,520192,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x59200000
1 madvise(0x59200000,520192,MADV_DONTFORK) = 0
1 munmap(0x59240000,258048) = 0
1 munmap(0x5923b000,20480) = 0

The end of the equivalent but much shorter loop on aarch64 looks like:

1 mremap(140484158312448,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
1 mremap(140484158308352,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
1 mremap(140484158304256,4096,8192,0,0,0) = -1 errno=12 (Out of memory)
1 mremap(140484158300160,4096,8192,0,0,0) = -1 errno=14 (Bad address)
1 mmap(NULL,16384,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANONYMOUS,-1,0) = 0x00007fc50452c000
1 mmap(0x00007fc509c40000,268693504,PROT_NONE,MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE,-1,0) = 0x00007fc4f44ed000
1 madvise(0x00007fc4f44ed000,268693504,MADV_DONTFORK) = 0
1 munmap(0x00007fc4f44ed000,77824) = 0
1 munmap(0x00007fc504500000,180224) = 0

Full log output:
logs.zip

[isheludko]

I tried installing Alpine linux to Arm32/Aarch64 qemu machines in a hope that I'll be able to run gdb there but node worked just fine.

Honestly, I'm out of ideas about how to proceed with this issue without being able to run a debugger.

[rotu]

This doesn't seem to loop forever on my Windows x64 PC. It does perform lots of mremaps such that time node -p 42 takes 7.5 seconds on qemu-arm versus 1.0 seconds on qemu-aarch64.

Today this does seem to loop forever on my Windows x64 PC, and the mremap first does seem to loop into the negatives again.