Crash with FeatureIndex::addFeature while Map::queryPointAnnotations

[tobrun]

When looking to reproduce #8202 / #8275 by constantly calling Map::queryPointAnnotation(), I was able to produce the following symbolicated stacktrace:

********** Crash dump: **********
Build fingerprint: 'google/angler/angler:7.1.2/NPG05F/3729850:user/release-keys'
pid: 25377, tid: 25377, name: pboxsdk.testapp  >>> com.mapbox.mapboxsdk.testapp <<<
signal 7 (SIGBUS), code 1 (BUS_ADRALN), fault addr 0x1000d
Stack frame #00 pc 0015d0bc  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >::__get_long_size() const at /Users/Nurbot/mapbox-gl-native/mason_packages/osx-x86_64/android-ndk/arm-v7-9-r13b/bin/../lib/gcc/arm-linux-androideabi/4.9.x/../../../../include/c++/4.9.x/string:1797
Stack frame #01 pc 0015d015  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine AnnotationTileFeature at /Users/Nurbot/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../src/mbgl/annotation/annotation_tile.hpp:27
Stack frame #02 pc 0016a461  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::FeatureIndex::addFeature(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::IndexedSubfeature const&, mbgl::GeometryCoordinates const&, std::experimental::fundamentals_v1::optional<std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > const&, mbgl::GeometryTileData const&, mbgl::CanonicalTileID const&, mbgl::style::Style const&, float, float) const at /Users/Nurbot/mapbox-gl-native/src/mbgl/geometry/feature_index.cpp:113 (discriminator 1)
Stack frame #03 pc 0016a363  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::FeatureIndex::query(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::GeometryCoordinates const&, float, double, double, std::experimental::fundamentals_v1::optional<std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > const&, mbgl::GeometryTileData const&, mbgl::CanonicalTileID const&, mbgl::style::Style const&, mbgl::CollisionTile const*) const at /Users/Nurbot/mapbox-gl-native/src/mbgl/geometry/feature_index.cpp:90
Stack frame #04 pc 001d2075  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::GeometryTile::queryRenderedFeatures(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::GeometryCoordinates const&, mbgl::TransformState const&, std::experimental::fundamentals_v1::optional<std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > const&) at /Users/Nurbot/mapbox-gl-native/src/mbgl/tile/geometry_tile.cpp:148
Stack frame #05 pc 000b8e73  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::style::Source::Impl::queryRenderedFeatures(mbgl::style::QueryParameters const&) const at /Users/Nurbot/mapbox-gl-native/src/mbgl/style/source_impl.cpp:246
Stack frame #06 pc 000bd071  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::style::Style::queryRenderedFeatures(mbgl::style::QueryParameters const&) const at /Users/Nurbot/mapbox-gl-native/src/mbgl/style/style.cpp:523
Stack frame #07 pc 0009d5a9  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::Map::queryRenderedFeatures(mapbox::geometry::box<double> const&, std::experimental::fundamentals_v1::optional<std::__ndk1::vector<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::allocator<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > > > > const&) at /Users/Nurbot/mapbox-gl-native/src/mbgl/map/map.cpp:826 (discriminator 1)
Stack frame #08 pc 0009d6d3  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::Map::queryPointAnnotations(mapbox::geometry::box<double> const&) at /Users/Nurbot/mapbox-gl-native/src/mbgl/map/map.cpp:840 (discriminator 3)
Stack frame #09 pc 00073283  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine mbgl::android::NativeMapView::queryPointAnnotations(_JNIEnv&, jni::Object<mbgl::android::RectF>) at /Users/Nurbot/mapbox-gl-native/platform/android/src/native_map_view.cpp:704 (discriminator 2)
Stack frame #10 pc 00088193  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine operator()<jni::jobject *, jni::jobject *> at /Users/Nurbot/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:59
Stack frame #11 pc 00525705  /data/app/com.mapbox.mapboxsdk.testapp-2/oat/arm/base.odex (offset 0x4fc000)

Been trying to produce this with a core gl test, but so far no luck.

[ncryu2]

I don't know this is the same problem, but crash on ios v3.4.1

malloc: *** mach_vm_map(size=4506811913453568) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
libc++abi.dylib: terminating with uncaught exception of type std::bad_alloc: std::bad_alloc

[ivovandongen]

This seems to be a duplicate of #8202 and can be reproduced fairly consistently on some devices (Moto X 2nd generation) with #8202 (comment).

There are two issues that are causing crashes. One is that the GeometryTileData member of GeometryTile passed to FeatureIndex::queryRenderedFeatures is invalid sometimes under heavy load. Reordering the members of GeometryTile so that data (GeometryTileData) is destructed after the FeatureIndex resolves this crash:

********** Crash dump: **********
Build fingerprint: 'motorola/victara_reteu/victara:5.0/LXE22.46-19/18:user/release-keys'
pid: 12444, tid: 12444, name: pboxsdk.testapp  >>> com.mapbox.mapboxsdk.testapp <<<
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x8
Stack frame #00 pc 00312dd8  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine std::__ndk1::pair<std::__ndk1::__hash_iterator<std::__ndk1::__hash_node<std::__ndk1::__hash_value_type<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mbgl::AnnotationTileLayer>, void*>*>, bool> std::__ndk1::__hash_table<std::__ndk1::__hash_value_type<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mbgl::AnnotationTileLayer>, std::__ndk1::__unordered_map_hasher<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::__hash_value_type<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mbgl::AnnotationTileLayer>, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, true>, std::__ndk1::__unordered_map_equal<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::__hash_value_type<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mbgl::AnnotationTileLayer>, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, true>, std::__ndk1::allocator<std::__ndk1::__hash_value_type<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, mbgl::AnnotationTileLayer> > >::__emplace_unique_key_args<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, mbgl::AnnotationTileLayer> const&>(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, mbgl::AnnotationTileLayer> const&) at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/__hash_table:1990
Stack frame #01 pc 00312bbf  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine unordered_map at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/unordered_map:1250
Stack frame #02 pc 00335195  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine _ZN6mapbox4util6detail17binary_dispatcherIN4mbgl5style15FilterEvaluatorIZNKS4_6FilterclINS3_19GeometryTileFeatureEEEbRKT_EUlSB_E_E10ComparatorIZNKSD_5equalERKNS_8geometry5valueESI_EUlSB_RKT0_E_EESG_bJNSF_12null_value_tEbyxdNSt6__ndk112basic_stringIcNSP_11char_traitsIcEENSP_9allocatorIcEEEENS0_17recursive_wrapperINSP_6vectorISG_NST_ISG_EEEEEENSW_INSP_13unordered_mapISV_SG_NSP_4hashISV_EENSP_8equal_toISV_EENST_INSP_4pairIKSV_SG_EEEEEEEEEE11apply_constESI_SI_OSN_ at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/variant/1.1.4/include/mapbox/variant.hpp:455
Stack frame #03 pc 00334eef  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine _ZNK4mbgl5style15FilterEvaluatorIZNKS0_6FilterclINS_19GeometryTileFeatureEEEbRKT_EUlS7_E_EclERKNS0_12EqualsFilterE at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../include/mbgl/style/filter_evaluator.hpp:34
Stack frame #04 pc 00439e47  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine __hash_table at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/__hash_table:993
Stack frame #05 pc 001735dd  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine std::__ndk1::__tree_node_destructor<std::__ndk1::allocator<std::__ndk1::__tree_node<std::__ndk1::__value_type<mbgl::OverscaledTileID, std::__ndk1::unique_ptr<mbgl::Tile, std::__ndk1::default_delete<mbgl::Tile> > >, void*> > >::operator()(std::__ndk1::__tree_node<std::__ndk1::__value_type<mbgl::OverscaledTileID, std::__ndk1::unique_ptr<mbgl::Tile, std::__ndk1::default_delete<mbgl::Tile> > >, void*>*) at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/__tree:756
Stack frame #06 pc 00181299  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine bool std::__ndk1::operator==<std::__ndk1::allocator<char> >(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&, std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const&) at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/string:3826
Stack frame #07 pc 0012e6fb  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine bool std::__ndk1::operator!=<mapbox::geometry::feature<double>*>(std::__ndk1::__wrap_iter<mapbox::geometry::feature<double>*> const&, std::__ndk1::__wrap_iter<mapbox::geometry::feature<double>*> const&) at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/iterator:1368
Stack frame #08 pc 0012ea85  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine std::__ndk1::vector<unsigned int, std::__ndk1::allocator<unsigned int> >::push_back(unsigned int const&) at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/vector:1589
Stack frame #09 pc 000bd2cb  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine __optional_storage at /Users/ivo/Library/Android/sdk/ndk-bundle/sources/cxx-stl/llvm-libc++/include/experimental/optional:214
Stack frame #10 pc 000f8ffd  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine _ZN3jni16MakeNativeMethodIZNS_17NativeMethodMakerIMZNS_30NativePeerMemberFunctionMethodIMN4mbgl7android13NativeMapViewEFNS_5ArrayINS_6ObjectINS4_7FeatureEEEvEER7_JNIEnvffNS6_INS7_INS_9StringTagEEEvEENS6_INS7_INS_9ObjectTagEEEvEEEXadL_ZNS5_29queryRenderedFeaturesForPointESC_ffSF_SI_EEEclIS5_S5_vEEDaRKNS_5FieldIT0_xEEEUlSC_NS7_IS5_EEffSF_SI_E_KFSA_SC_ST_ffSF_SI_EJEEclISU_EESN_PKcRKT_EUlPSB_PNS_7jobjectEffPNS_6jarrayIS15_EES19_E_EEDaS10_S10_S13_PNSt6__ndk19enable_ifIXsr3std8is_classIS11_EE5valueEvE4typeE at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:53 (discriminator 362)
Stack frame #11 pc 000f8f63  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine jni::Field jni::NativeMethodMaker<jni::Array<jni::Object<mbgl::android::Feature>, void> (auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}::*)(_JNIEnv&, {lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>) const>::operator()<auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}>(char const*, auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1} const&) at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:115
Stack frame #12 pc 000f8f03  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine jni::Field jni::NativeMethodMaker<jni::Array<jni::Object<mbgl::android::Feature>, void> (auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}::*)(_JNIEnv&, {lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>) const>::operator()<auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1}>(char const*, auto jni::NativePeerMemberFunctionMethod<jni::Array<jni::Object<mbgl::android::Feature>, void> (mbgl::android::NativeMapView::*)(_JNIEnv&, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>), &mbgl::android::NativeMapView::queryRenderedFeaturesForPoint>::operator()<mbgl::android::NativeMapView, mbgl::android::NativeMapView, void>(jni::Field<mbgl::android::NativeMapView, long long> const&)::{lambda(_JNIEnv&, jni::Object<mbgl::android::NativeMapView>, float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)#1} const&) at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:107
Stack frame #13 pc 000f90a5  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine jni::TypeSignature<jni::Array<jni::Object<mbgl::android::Feature>, void> (float, float, jni::Array<jni::Object<jni::StringTag>, void>, jni::Array<jni::Object<jni::ObjectTag>, void>)>::operator()() const at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/type_signature.hpp:62 (discriminator 242)
Stack frame #14 pc 000f905b  /data/app/com.mapbox.mapboxsdk.testapp-2/lib/arm/libmapbox-gl.so: Routine _ZN3jni16MakeNativeMethodIZNS_17NativeMethodMakerIMZNS_30NativePeerMemberFunctionMethodIMN4mbgl7android13NativeMapViewEFNS_5ArrayINS_6ObjectINS4_7FeatureEEEvEER7_JNIEnvffNS6_INS7_INS_9StringTagEEEvEENS6_INS7_INS_9ObjectTagEEEvEEEXadL_ZNS5_29queryRenderedFeaturesForPointESC_ffSF_SI_EEEclIS5_S5_vEEDaRKNS_5FieldIT0_xEEEUlSC_NS7_IS5_EEffSF_SI_E_KFSA_SC_ST_ffSF_SI_EJEEclISU_EESN_PKcRKT_EUlPSB_PNS_7jobjectEffPNS_6jarrayIS15_EES19_E_EEDaS10_S10_S13_PNSt6__ndk19enable_ifIXsr3std8is_classIS11_EE5valueEvE4typeE at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:53 (discriminator 365)
Stack frame #15 pc 00171e2f  /data/dalvik-cache/arm/data@app@com.mapbox.mapboxsdk.testapp-2@base.apk@classes.dex

The other problem seems to be that the indices in FeatureIndex passed to AnnotationTileLayer::getFeature() are sometimes out of bounds under high load (verified with an assertion). Which leads to:

********** Crash dump: **********
Build fingerprint: 'motorola/victara_reteu/victara:5.0/LXE22.46-19/18:user/release-keys'
pid: 30403, tid: 30403, name: pboxsdk.testapp  >>> com.mapbox.mapboxsdk.testapp <<<
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
Stack frame #00 pc 0003c22c  /system/lib/libc.so (tgkill+12)
Stack frame #01 pc 0001667d  /system/lib/libc.so (pthread_kill+52)
Stack frame #02 pc 0001728f  /system/lib/libc.so (raise+10)
Stack frame #03 pc 00013ae1  /system/lib/libc.so (__libc_android_abort+36)
Stack frame #04 pc 00012264  /system/lib/libc.so (abort+4)
Stack frame #05 pc 00014dab  /system/lib/libc.so (__libc_fatal+16)
Stack frame #06 pc 00013b65  /system/lib/libc.so (__assert2+20)
Stack frame #07 pc 0030f5a5  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::AnnotationTileLayer::getFeature(unsigned int) const at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../src/mbgl/annotation/annotation_tile.hpp:49 (discriminator 2)
Stack frame #08 pc 00331ccd  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::FeatureIndex::addFeature(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::IndexedSubfeature const&, mbgl::GeometryCoordinates const&, mbgl::QueryOptions const&, mbgl::GeometryTileData const&, mbgl::CanonicalTileID const&, mbgl::style::Style const&, float, float) const at /Users/ivo/git/mapbox-gl-native/src/mbgl/geometry/feature_index.cpp:116
Stack frame #09 pc 00331a27  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::FeatureIndex::query(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::GeometryCoordinates const&, float, double, double, mbgl::QueryOptions const&, mbgl::GeometryTileData const&, mbgl::CanonicalTileID const&, mbgl::style::Style const&, mbgl::CollisionTile const*) const at /Users/ivo/git/mapbox-gl-native/src/mbgl/geometry/feature_index.cpp:93
Stack frame #10 pc 00433703  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::GeometryTile::queryRenderedFeatures(std::__ndk1::unordered_map<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> >, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > >, std::__ndk1::hash<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::equal_to<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > >, std::__ndk1::allocator<std::__ndk1::pair<std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char> > const, std::__ndk1::vector<mapbox::geometry::feature<double>, std::__ndk1::allocator<mapbox::geometry::feature<double> > > > > >&, mbgl::GeometryCoordinates const&, mbgl::TransformState const&, mbgl::QueryOptions const&) at /Users/ivo/git/mapbox-gl-native/src/mbgl/tile/geometry_tile.cpp:149
Stack frame #11 pc 0016f88d  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::style::Source::Impl::queryRenderedFeatures(mapbox::geometry::line_string<double, std::__ndk1::vector> const&, mbgl::TransformState const&, mbgl::QueryOptions const&) const at /Users/ivo/git/mapbox-gl-native/src/mbgl/style/source_impl.cpp:252
Stack frame #12 pc 0017d561  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::style::Style::queryRenderedFeatures(mapbox::geometry::line_string<double, std::__ndk1::vector> const&, mbgl::TransformState const&, mbgl::QueryOptions const&) const at /Users/ivo/git/mapbox-gl-native/src/mbgl/style/style.cpp:526
Stack frame #13 pc 0012e313  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::Map::queryRenderedFeatures(mapbox::geometry::box<double> const&, mbgl::QueryOptions const&) at /Users/ivo/git/mapbox-gl-native/src/mbgl/map/map.cpp:825 (discriminator 1)
Stack frame #14 pc 0012e69d  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::Map::queryPointAnnotations(mapbox::geometry::box<double> const&) at /Users/ivo/git/mapbox-gl-native/src/mbgl/map/map.cpp:841
Stack frame #15 pc 000bcee3  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine mbgl::android::NativeMapView::queryPointAnnotations(_JNIEnv&, jni::Object<mbgl::android::RectF>) at /Users/ivo/git/mapbox-gl-native/platform/android/src/native_map_view.cpp:714 (discriminator 1)
Stack frame #16 pc 000f8c15  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine operator() at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:245 (discriminator 325)
Stack frame #17 pc 000f8b7b  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine operator() at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:112 (discriminator 187)
Stack frame #18 pc 000f8b1b  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine __invoke at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:110
Stack frame #19 pc 000f8cbd  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine operator()<jni::jobject *, jni::jobject *> at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:59
Stack frame #20 pc 000f8c73  /data/app/com.mapbox.mapboxsdk.testapp-1/lib/arm/libmapbox-gl.so: Routine __invoke<jni::jobject *, jni::jobject *> at /Users/ivo/git/mapbox-gl-native/build/android-arm-v7-9/Debug/../../../mason_packages/headers/jni.hpp/3.0.0/include/jni/native_method.hpp:55
Stack frame #21 pc 00171e2f  /data/dalvik-cache/arm/data@app@com.mapbox.mapboxsdk.testapp-1@base.apk@classes.dex

This seems to be the same as #8289 (comment) on iOS.

cc @kkaefer

[ivovandongen]

Debugging some more, it seem we might be accessing the GeometryTileData from multiple threads concurrently:

03-08 22:31:58.813 4382-7575/com.mapbox.mapboxsdk.testapp I/mbgl: {Worker 2}[General]: AnnotationTileLayer::getFeature(32)
03-08 22:31:58.813 4382-7575/com.mapbox.mapboxsdk.testapp I/mbgl: {Worker 2}[General]: AnnotationTileLayer::getFeature(33)
03-08 22:31:58.814 4382-7575/com.mapbox.mapboxsdk.testapp I/mbgl: {Worker 2}[General]: AnnotationTileLayer::getFeature(34)
03-08 22:31:58.814 4382-7575/com.mapbox.mapboxsdk.testapp I/mbgl: {Worker 2}[General]: AnnotationTileLayer::getFeature(35)
03-08 22:31:58.843 4382-4382/com.mapbox.mapboxsdk.testapp I/mbgl: {pboxsdk.testapp}[General]: AnnotationTileLayer::getFeature(0)
                                                                  
                                                                  --------- beginning of crash
03-08 22:31:58.843 4382-4382/com.mapbox.mapboxsdk.testapp A/libc: ../../../src/mbgl/annotation/annotation_tile.hpp:51: virtual std::unique_ptr<GeometryTileFeature> mbgl::AnnotationTileLayer::getFeature(std::size_t) const: assertion "features.size() > 0" failed
03-08 22:31:58.843 4382-4382/com.mapbox.mapboxsdk.testapp A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 4382 (pboxsdk.testapp)

(Note the "Worker 2" and "pboxsdk.testapp" (main) thread)

[jfirebaugh]

GeometryTileData, the class, is used on both the main thread and on worker threads, but individual instances aren't accessed concurrently.

I think the issue here is somehow feature indexes are getting into the FeatureIndex that are out of bounds for the vector of features held by the AnnotationTileLayer. Not sure how that happens though.

[ivovandongen]

Digging into why the index in FeatureIndex doesn't match up with with the indices in AnnotationTileLayer it seems that there is a moment between AnnotationManager::updateAnnotation and redoing the layout where this might occur.

[ivovandongen]

The actual problem lies in a mismatch between the GeometryTileData and CollisionTile on GeometryTile and stems from the fact that both are not updated atomically.

The updating of both happens in GeometryTileWorker::redoLayout() where first GeometryTile#onLayout is called (which sets GeometryTileData) asynchronously and then GeometryTile#onPlacement (which sets the CollisionTile) after GeometryTileWorker#attemptPlacement completes.

In GeometryTile::queryRenderedFeatures the GeometryTileData and CollisionTile are passed to FeatureIndex::query. Here the CollisionTile is used to lookup the IndexedSubFeatures which are used in FeatureIndex::addFeature to get the actual Feature from the GeometryTileLayer retrieved from the GeometryTileData

This goes wrong when the worker has called GeometryTile::onLayout after an Annotation is updated and crosses tile boundaries, but GeometryTile::onPlacement is not yet called when GeometryTile::queryRenderedFeatures is called. In this case the new GeometryTileData and the old CollisionTile are passed to FeatureIndex::query possibly causing an out of bounds error in AnnotationTileLayer::getFeature because the bounds of the vector are not checked (assumed to be correct)

[jfirebaugh]

Nice work tracking that down @ivovandongen. It hadn't occurred to me that the issue could be the GeometryTileData and CollisionTile getting out of sync. Will comment on the PR with some further thoughts.