Planet.osm with OSRM on a 256Go RAM machine not working

[Guipa]

Hi,

We are facing a dilemna .. We have contracted for a 256Go RAM server with 16 threads and 600 Go HD SAS (with Ubuntu 12.4 LTS) in order to run OSRM with Planet.osm file.
All extract and prepare process seem to be working properly. When we launch osrm-routed, we have a message "could not be locked to ram" but it continues until waiting for request. When we put a request, OSRM crashes with a segmentation fault message. When we look at the files generated during the prepare process, it appears that the planet hsgr file is only 17 Go (it should be 32Go), all others files have a correct size. We have re-execute the extract and prepare process 3 times and still have the same results (hsgr file of 17 Go and "could not be locked to RAM"). Our stxxl file mentions a 250 Go virtual Drive.

2 weeks ago, we have execute extract, prepare on a big Amazon EC2 server with 128 Go RAM, running also with Ubuntu 12.4) and it was working properly (osrm-routed working fine and all files have the correct size). The planet.osm.pbf file is the same on both amazon and our server.

Any idea of what is going on ?

Thanks

Guip

[DennisOSRM]

data files appear to be broken. Rebuild the source from latest develop branch, delete all data files, and regenerate all data files.

Also, when it complains that the process cannot be locked into RAM, then your machine has not enough free RAM. Make sure that you are actually on a 64bit Linux kernel. uname -a should say something about x86_64

[DennisOSRM]

Oh, and check free disk space with df -h

[DennisOSRM]

Any news on this?

[Guipa]

Hi denis,
Sorry for not having replied but we have been on holidays since. Coming
back next week, we will try with the dev branch and tell you if it is
working. Thanks. Guillaume
Le 17 juil. 2013 07:25, "Dennis Luxen" notifications@github.com a écrit :

Any news on this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/661#issuecomment-21106640
.

[Guipa]

Hi Dennis,

We have installed the develop branch of osrm, download a new planet.osm.pbf file. No error messages for extract and prepare. When we launch routed, we still have the could not Locked to ram message. Nearest command works and gives results but when we do a viaroute command, it can't find a route and osrm crash with segment fault message.
We have done the command to inquire about 64bits and free disk space : the machine is 64bits and has 250 go disk space. The available ram with osrm-routed launched is 110 go (the machine has 256 go ram). When we extract and prepare a smallest pbf file (a region), osrm is working great.

What could be the problem ?

Thanks

Sent from my iPad

On 17 juil. 2013, at 07:25, Dennis Luxen notifications@github.com wrote:

Any news on this?

—
Reply to this email directly or view it on GitHub.

[DennisOSRM]

Can you start in gdb and provide a stack trace?

[Guipa]

We have started in gdb extract and prepare with full backtrace. Program seems to go through normally as we do not have any backtrace logs ... Maybe we do something wrong with the bt command ...

Sent from my iPad

On 26 juil. 2013, at 11:14, Dennis Luxen notifications@github.com wrote:

Can you start in gdb and provide a stack trace?

—
Reply to this email directly or view it on GitHub.

[DennisOSRM]

Any updates?

[Guipa]

Hi Dennis,

As it was still not working on the 256 Go RAM provided by our hosting company, we have change for another server (with less RAM : 128 Go) with a new hosting company. Things are working properly now. I guess the 256Go RAM machine we had before was not exactly as good as it should be …

Le 16 déc. 2013 à 13:46, Dennis Luxen notifications@github.com a écrit :

Any updates?

—
Reply to this email directly or view it on GitHub.

[DennisOSRM]

Alright, thanks for the update. Closing here.

[kinju]

I have this problem too ("[warn] Process ./osrm-routed could not be locked to RAM")

• My Server :
• Linux geoptima-srv 3.11.0-18-generic contractor is now slower #32~precise1-Ubuntu SMP Thu Feb 20 17:52:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
• 32GO RAM
• 250GO SSD HD

I built OSRM develop branch from today.

• cmake :
  kin@myserver:~/Project-OSRM/build$ cmake -DCMAKE_BUILD_TYPE=Debug ..
  -- The C compiler identification is GNU
  -- The CXX compiler identification is GNU
  -- Check for working C compiler: /usr/bin/gcc
  -- Check for working C compiler: /usr/bin/gcc -- works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++
  -- Check for working CXX compiler: /usr/bin/c++ -- works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Building on a 64 bit system
  -- Configuring OSRM in debug mode
  -- adding profiling flags
  -- Performing Test HAS_LTO_FLAG
  -- Performing Test HAS_LTO_FLAG - Success
  -- Performing Test HAS_LTO_PARTITION_FLAG
  -- Performing Test HAS_LTO_PARTITION_FLAG - Success
  -- Boost version: 1.46.1
  -- Found the following Boost libraries:
  -- date_time
  -- filesystem
  -- iostreams
  -- program_options
  -- regex
  -- system
  -- thread
  -- Looking for include files CMAKE_HAVE_PTHREAD_H
  -- Looking for include files CMAKE_HAVE_PTHREAD_H - found
  -- Looking for pthread_create in pthreads
  -- Looking for pthread_create in pthreads - not found
  -- Looking for pthread_create in pthread
  -- Looking for pthread_create in pthread - found
  -- Found Threads: TRUE
  -- Could NOT find Lua52 (missing: LUA_LIBRARIES LUA_INCLUDE_DIR)
  -- Found Lua51: /usr/lib/x86_64-linux-gnu/liblua5.1.so;/usr/lib/x86_64-linux-gnu/libm.so
  -- Looking for LuaJIT 5.1
  -- Found LUAJIT: /usr/lib/x86_64-linux-gnu/libluajit-5.1.so;/usr/lib/x86_64-linux-gnu/libm.so
  -- Found LuaJIT: /usr/lib/x86_64-linux-gnu/libluajit-5.1.so
  -- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so
  -- Looking for Luabind...
  -- Found Luabind: /usr/lib/libluabind.so
  -- Found Luabind: /usr/lib/libluabind.so
  -- Looking for STXXL...
  -- Found STXXL: /usr/lib/libstxxl.so
  -- Found STXXL: /usr/lib/libstxxl.so
  -- Looking for OSMPBF...
  -- Found OSMPBF: /usr/lib/libosmpbf.a
  -- Found OSMPBF: /usr/lib/libosmpbf.a
  -- Found PROTOBUF: /usr/lib/libprotobuf.so
  -- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so
  -- Looking for BZ2_bzCompressInit in /usr/lib/x86_64-linux-gnu/libbz2.so
  -- Looking for BZ2_bzCompressInit in /usr/lib/x86_64-linux-gnu/libbz2.so - found
  -- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.3.4")
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/user/Project-OSRM/build
• Execution :
  kin@myserver:~/Project-OSRM/build$ gdb ./osrm-routed
  GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
  Copyright (C) 2012 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law. Type "show copying"
  and "show warranty" for details.
  This GDB was configured as "x86_64-linux-gnu".
  For bug reporting instructions, please see:
  http://bugs.launchpad.net/gdb-linaro/...
  Reading symbols from /home/kin/Project-OSRM/build/osrm-routed...done.
  (gdb) r europe-latest.osm.pbf
  Starting program: /home/kin/Project-OSRM/build/osrm-routed europe-latest.osrm
  warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
  [Thread debugging using libthread_db enabled]
  Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
  [warn] Process /home/kin/Project-OSRM/build/osrm-routed could not be locked to RAM
  [info] starting up engines, v0.3.7-290-g1222aa6, compiled at Apr 10 2014, 16:44:19
  [info] HSGR file: "europe-latest.osrm.hsgr"
  [debug] Nodes file: "europe-latest.osrm.nodes"
  [debug] Edges file: "europe/europe-latest.osrm.edges"
  [debug] RAM file: "europe/europe-latest.osrm.ramIndex"
  [debug] Index file: "europe/europe-latest.osrm.fileIndex"
  [debug] Names file: "europe/europe-latest.osrm.names"
  [debug] Timestamp file: "europe/europe-latest.osrm.timestamp"
  [debug] Threads: 8
  [debug] IP address: 0.0.0.0
  [debug] IP port: 5000
  [info] loading graph data
  [info] loading graph from europe-latest.osrm.hsgr
  [info] loaded 325499874 nodes and 894255257 edges
  . 10% . 20% . 30% . 40% . 50% . 60% . 70% . 80% . 90% . 100%
  [info] Data checksum is 1068992215
  [info] loading egde information
  [debug] Loading node data
  [debug] Loading edge data
  [info] loading r-tree
  [info] loading timestamp
  [info] loading street names
  [info] loaded plugin: hello
  [info] loaded plugin: locate
  [info] loaded plugin: nearest
  [info] loaded plugin: timestamp
  [info] loaded plugin: viaroute
  [info] http 1.1 compression handled by zlib version 1.2.3.4
  [New Thread 0x7ffd08ab8700 (LWP 12246)]
  [info] running and waiting for requests
  [New Thread 0x7ffd03fff700 (LWP 12247)]
  [New Thread 0x7ffd037fe700 (LWP 12248)]
  [New Thread 0x7ffd02ffd700 (LWP 12249)]
  [New Thread 0x7ffd027fc700 (LWP 12250)]
  [New Thread 0x7ffd01ffb700 (LWP 12251)]
  [New Thread 0x7ffd017fa700 (LWP 12252)]
  [New Thread 0x7ffd00ff9700 (LWP 12253)]
  [New Thread 0x7ffcebfff700 (LWP 12254)]

I have several machines and which are using OSRM and I update OSRM every weeks. It display this warning only since 2/3 months, even on a small osm.pbf file (my county - 150mb).

Let me know if you need more info.

Thanks

[DennisOSRM]

The warning is non-critical. It basically says that the OS refused to turn of swapping for the process in case memory availability becomes low. Most of the times this is caused by more RAM being in use already than is requested.