Bad address issue when starting

[jbfarez]

Overview

Hello there!

I'm landing here because I'm facing an issue when starting the backend and I just found nothing related here. When I'm trying to start the service, after several minutes, it suddenly crash with following message:

"[error] Problem reading from file: /data/europe-latest.osrm.mldgr : /mld/multilevelgraph/edge_array (possible cause: "Bad address") (at include/storage/tar.hpp:42)"

It's a serious issue, it's blocking a release, I really hope to find some help here.

Thanks!

Details

Environment

• OSRM version: v5.17.2
• OS: Debian
• OS version: 8.7
• Docker version: 18.06.3-ce
• Kernel release: 4.9.0-0.bpo.8
• FD limit: 104000

How I build the base:

# Extract
docker run -t -v /var/lib/osrm:/data osrm/osrm-backend:v5.17.2 osrm-extract -p /opt/car.lua /data/europe-latests.osm.pbf

# Partition
docker run -t -v /var/lib/osrm:/data osrm/osrm-backend:v5.17.2 osrm-partition /data/europe-latest.osrm

# Customize
docker run -t -v /var/lib/osrm:/data osrm/osrm-backend:v5.17.2 osrm-customize /data/europe-latest.osrm

How I start the service

I'm using a systemd unit to start the service with the following command:

/usr/bin/docker run --restart=always -p 5000:5000 --name=osrm -v /var/lib/osrm:/data osrm/osrm-backend:v5.17.2 osrm-routed \
                          --memory_file /osrm.dat \
                          --max-table-size=99999999 \
                          --algorithm mld /data/europe-latest.osrm

[danpat]

@jbfarez The first thing I'd try is upgrading to the most recent release - I don't recall any bugs like this that were fixed, but 🤷‍♂️ .

You would need to re-process your data with the newer OSRM version.

I'd also verify that all your processing steps completed without error. osrm-routed's error messages can sometimes be unclear - it might be that your datafiles are incomplete, and the error is a touch misleading.

[jbfarez]

Hey @danpat.

Thanks for your answer! So, we will try to upgrade ASAP but it's a production service so, not that easy at this time.
In the meantime, I've found a workaround which consists to remove the --mermory-file flag and to setup a large SWAP space (30Gb).

So my 1st thoughts around this issue is that it probably comes from the storage capabilities. In this case, the service was running on an AWS m5.large (2vcpus, 8Gb RAM...) with 150Gb of storage on an EBS volume (gp2 based which mean around 450 iops). Thus, my guess is that during the startup when OSRM is trying to write inside the "memory file" on the disk, it reach some hard limits and become throttled by AWS.

This theory needs to be confirmed but this is the most likely issue. If - at some point - I get more information about this issue, I'll let you know here.

Hope this can help and I think it can be closed.

[danpat]

Ah-ha. I missed the --memory-file. Well, another good reason to upgrade to OSRM 5.22 would be that it now supports mmap()-ing files directly, no need for that /osrm.dat file. Support for doing that landed with this PR: #5242