osrm-extract Error bad_alloc

[ghost]

Hi,

when using osrm-extract (after building from source) I get the following error message.

[info] Using script profiles/car.lua
[info] Input file: germany-latest.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[info] Parsing in progress..
[info] Using profile api version 2
[info] input file generated by osmium/1.5.1
[info] timestamp: 2017-09-04T20:43:02Z
[info] Found 3 turn restriction tags:
[info] motorcar
[info] motor_vehicle
[info] vehicle
[info] Using profile api version 2
[info] Using profile api version 2
[info] Using profile api version 2
TBB Warning: Exact exception propagation is requested by application but the linked library is built without support for it
terminate called after throwing an instance of 'tbb::captured_exception'
what(): std::bad_alloc
Aborted (core dumped)

I am using Ubuntu 16.04 and did everything exactly like described in the readme.

[ghost]

Okay, I solved the problem. There was just not enough RAM available. Sorry :S

[tds4u]

Just my two cents...

If you need to convert whole planet.pbf (2017-09-25) than you need at least

• car: >250 GB RAM
• bicycle: >280GB RAM
• foot: >300GB RAM

with real memory or swap space.

Prerequisites:

fallocate -l 64G /scripts/swap.swap
chmod 600 /scripts/swap.swap
mkswap /scripts/swap.swap
swapon /scripts/swap.swap
echo 20 > /proc/sys/vm/swappiness

Hardware:

Processor: Intel(R) Xeon(R) CPU E5-2630 v4
Threads: 40
Memory: 256GB DDR4 ECC RAM
Swap: 8GB as partition + 64GB with fallocate
Disk: /dev/sda2       1.8T  159G  1.6T  10% /
Profile 1: car.lua without problems
Profile 2: bicycle.lua with TBB exception => expand RAM via Swap
Profile 3: foot.lua with TBB exception => expand RAM via Swap

Work:

osrm-extract planet-latest.osm.pbf -p profiles/[car|bicycle|foot].lua
[info] Using script /usr/local/share/osrm/profiles/bicycle.lua
[info] Input file: planet-latest.osm.pbf
[info] Profile: bicycle.lua
[info] Threads: 40
[info] Parsing in progress..
[info] Using profile api version 2
[info] input file generated by planet-dump-ng 1.1.4
[info] timestamp: 2017-09-25T01:59:59Z
[info] Parse relations ...
...
a.s.o.

Success if no exception, error or similar.

[emiltin]

using swap instead of ram will be very very slow

[dooley]

We're running into the same problem today with OSRM 5.14.1 with foot.lua processing the planet file using a server with 256 GB RAM and 4 TB HDD.

Question: If we preprocess the planet by filter out i.e. buildings=, natural= and a lot of other objects, could that help? If yes, does anybody has a list of keys which is used by OSRM?

[javalver]

Hi,
I have the same problem.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.4 LTS
Release: 16.04
Codename: xenial
8 GB RAM
300 GB HD
4 processors
Trying to osrm-extract a 347MB size .pbf file, does not work in version 5.17.0.

Before, in a server running osrm-extract version 5.15.0 with 4 GB RAM and 100 GB HD, and it worked fine.
The command is:
osrm-extract /opt/osrm/data/map.osm.pbf -p /opt/osrm/osrm-backend/profiles/car.lua

I'm processing Central America file.

Any suggestions?

[danpat]

@javalver what(): std::bad_alloc indicates you have run out of RAM - make sure you actually have free memory on your system.

[javalver]

In fact, I have 6 GB free RAM and about 70 GB free swap.

[javalver]

As I've said before, before I hadn't that problem, even in another system with less RAM and HD.

[danpat]

@javalver Can you please provide a bit more information about the error you're getting? Please copy/paste the complete output here.

Note that OSRM 5.17 is not an official release, it sounds like you're building from master, so please also provide the gitsha for the exact code you're using.

[javalver]

root@osrm02:/opt/osrm/data# osrm-extract /opt/osrm/data/map.osm.pbf -p /opt/osrm/osrm-backend/profiles/car.lua
[info] Parsed 0 location-dependent features with 0 GeoJSON polygons
[info] Using script /opt/osrm/osrm-backend/profiles/car.lua
[info] Input file: map.osm.pbf
[info] Profile: car.lua
[info] Threads: 4
[info] Parsing in progress..
[info] input file generated by osmium/1.5.1
[info] timestamp: 2018-03-06T21:43:02Z
[info] Using profile api version 4
[info] Found 3 turn restriction tags:
[info] motorcar
[info] motor_vehicle
[info] vehicle
[info] Parse relations ...
[info] Parse ways and nodes ...
[info] Using profile api version 4
[info] Using profile api version 4
[info] Using profile api version 4
TBB Warning: Exact exception propagation is requested by application but the linked library is built without support for it
terminate called after throwing an instance of 'tbb::captured_exception'
what(): std::bad_alloc
Aborted (core dumped)

I'm getting version 5.17 directly from git clone https://github.com/Project-OSRM/osrm-backend.git

[danpat]

@javalver You are definitely running out of RAM.

Is the map.osm.pbf file you're using downloadable from somewhere? Are you generating it yourself, or can I download it from Geofabrik or some other source?

[javalver]

I'm using this command to get .pbf file:

wget -O map.osm.pbf http://download.geofabrik.de/central-america-latest.osm.pbf

[danpat]

@javalver I just tried with that file and the same gitsha - it works fine for me and uses 2.7GB of RAM.

/usr/bin/time -l ./osrm-extract -p ../profiles/car.lua ../central-america-latest.osm.pbf
....
      200.51 real       530.58 user        49.99 sys
2730561536  maximum resident set size

Try running the same command I did /usr/bin/time -l .... and see how much memory it needs before it dies.

[javalver]

I've run that command and got the next results:

Command terminated by signal 6
Command being timed: "osrm-extract /opt/osrm/data/map.osm.pbf -p /opt/osrm/osrm-backend/profiles/car.lua"
User time (seconds): 260.47
System time (seconds): 15.82
Percent of CPU this job got: 370%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:14.65
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 1437132
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 4019164
Voluntary context switches: 80572
Involuntary context switches: 30788
Swaps: 0
File system inputs: 0
File system outputs: 8
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

The server has 8 GB RAM and 300 GB HD

[danpat]

@javalver there is some problem with your system.

Maximum resident set size (kbytes): 1437132

Your program quit very quickly and didn't even allocate a lot of RAM, even though it received an OOM error.

If you're running inside Docker or some other container, please ensure that you haven't specified some very low memory limits for the container. Also check that the RSS ulimit for your process is unlimited.

Unfortunately, the problem looks like a configuration problem on your system, not something to do with OSRM itself - I'm probably not going to be able to help you much further, you could try a StackOverflow forum regarding Linux system configuration.

[javalver]

Hi everyone,
I've checked all the server configuration and didn't find any bad configuration. So, I've opted for migrating all the solution to Centos 7 platform, and, Eureka! Everything is working well now!
Many thanks for your advices.

[answerquest]

I'm getting the same problem. RAM consumption doesn't even go up too much, the program just quits.