Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lua scripting speedups #4147

Merged
merged 2 commits into from
Jun 13, 2017
Merged

Lua scripting speedups #4147

merged 2 commits into from
Jun 13, 2017

Conversation

danpat
Copy link
Member

@danpat danpat commented Jun 12, 2017

Issue

After a bit of conversation with @joto, it looks like there are some easy speedups we can make with our Lua handling code to make osrm-exract run a lot faster.

  1. Don't make copies of sol::function objects for every call - this is simply unnecessary, and using thread-local cached copies gives us a 12-13% speedup very easily.

  2. Make calling the node_function optional. If the profile property call_tagless_node_function is set to false, then the node_function won't get called if there are no tags on a node (you only need to call this in situations where you're doing coordinate lookups for nodes for some reason. The default profiles don't do this). Because there are more nodes than anything else, and most of them have no tags, this creates a speedup of 269 sec -> 135 seconds testing us-west-latest.osm.pbf.
    This change should be backwards compatible. For those with old scripts that don't include the new call_tagless_node_function profile property, it will default to the old behaviour of calling the node_function for every node. Users can add this property to their existing scripts to gain access to the speedup.

Tasklist

  • review
  • adjust for comments

… 12-13% speedup for lua processing right there.
Copy link
Contributor

@oxidase oxidase left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@danpat
Copy link
Member Author

danpat commented Jun 13, 2017

I did some additional testing with us-west-latest.osm.pbf. If I check for building tags on ways in C++, it cuts a further 11 seconds (8%) off the OSM parsing phase of osrm-extract. However, this seems a bit too special-casey and could possibly bite us on say, the foot profile. I'll ticket this for later optimization (checking tag white/blacklists in C++), but I think this PR represents the best bang-for-buck for now.

@danpat
Copy link
Member Author

danpat commented Jun 13, 2017

Note: processing planet.osm.pbf on an r3.8xlarge AWS instance currently spends 2700-3000 seconds doing OSM parsing/Lua script stuff. This change cuts that roughly in half, saving about 20-25 minutes when processing the whole planet.

Copy link
Member

@TheMarex TheMarex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, such simple changes!

@TheMarex TheMarex merged commit cd8fb82 into master Jun 13, 2017
@TheMarex TheMarex deleted the optional_lua_calls branch June 13, 2017 12:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants