-
-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🐛 FrameProcessor crash on Android #350
Comments
Huh? A null pointer de-reference error? Are you allocating something in your plugin? Please share the code of the frame processor as well as the frame processor plugin |
I'll see if I can replicate the issue with the camera example. |
Yep still occurs even if reproduced in example app (takes a lot longer to happen, like over 5 minutes):
ObjectDetectorFrameProcessorPlugin.java: package com.mrousavy.camera.example;
import com.facebook.react.bridge.WritableNativeMap;
import com.facebook.react.bridge.WritableNativeArray;
import com.google.android.gms.tasks.Task;
import com.google.android.gms.tasks.Tasks;
import com.google.android.gms.tasks.OnFailureListener;
import com.google.android.gms.tasks.OnSuccessListener;
import com.mrousavy.camera.frameprocessor.FrameProcessorPlugin;
import org.jetbrains.annotations.NotNull;
import android.os.Bundle;
import android.media.Image;
import android.graphics.Rect;
import android.graphics.Point;
import android.graphics.Bitmap;
import android.annotation.SuppressLint;
import androidx.annotation.NonNull;
import androidx.camera.core.ImageProxy;
import androidx.appcompat.app.AppCompatActivity;
import com.google.mlkit.common.model.LocalModel;
import com.google.mlkit.vision.common.InputImage;
import com.google.mlkit.vision.objects.DetectedObject;
import com.google.mlkit.vision.objects.ObjectDetection;
import com.google.mlkit.vision.objects.ObjectDetector;
// import com.google.mlkit.vision.objects.custom.CustomObjectDetectorOptions;
import com.google.mlkit.vision.objects.defaults.ObjectDetectorOptions;
import com.google.mlkit.vision.objects.defaults.PredefinedCategory;
import java.util.List;
import java.util.ArrayList;
import java.util.concurrent.ExecutionException;
public class ObjectDetectorFrameProcessorPlugin extends FrameProcessorPlugin {
ObjectDetectorOptions options =
new ObjectDetectorOptions.Builder()
.setDetectorMode(ObjectDetectorOptions.STREAM_MODE)
.enableMultipleObjects()
.enableClassification()
.build();
ObjectDetector objectDetector = ObjectDetection.getClient(options);
@Override
public Object callback(ImageProxy frame, @NotNull Object[] params) {
@SuppressLint("UnsafeOptInUsageError")
Image mediaImage = frame.getImage();
if (mediaImage != null) {
InputImage image = InputImage.fromMediaImage(mediaImage, frame.getImageInfo().getRotationDegrees());
Task<List<DetectedObject>> task = objectDetector.process(image);
try {
List<DetectedObject> results = Tasks.await(task);
WritableNativeArray array = new WritableNativeArray();
for (DetectedObject detectedObject : results) {
Rect boundingBox = detectedObject.getBoundingBox();
Integer trackingId = detectedObject.getTrackingId();
WritableNativeArray boundingBoxArray = new WritableNativeArray();
boundingBoxArray.pushDouble(boundingBox.top);
boundingBoxArray.pushDouble(boundingBox.left);
boundingBoxArray.pushDouble(boundingBox.bottom);
boundingBoxArray.pushDouble(boundingBox.right);
WritableNativeArray labelsArray = new WritableNativeArray();
for (DetectedObject.Label label : detectedObject.getLabels()) {
WritableNativeMap labelsMap = new WritableNativeMap();
labelsMap.putString("label", label.getText());
labelsMap.putDouble("confidence", label.getConfidence());
labelsArray.pushMap(labelsMap);
}
WritableNativeMap map = new WritableNativeMap();
map.putArray("boundingBox", boundingBoxArray);
map.putInt("trackingId", trackingId);
map.putArray("labels", labelsArray);
array.pushMap(map);
}
return array;
} catch (Exception e) {
e.printStackTrace();
}
}
return null;
}
ObjectDetectorFrameProcessorPlugin() {
super("object_detector_plugin");
}
} frameProcessor: const objectBounds = useSharedValue([0, 0, 0, 0]);
const frameProcessor = useFrameProcessor(
(frame) => {
'worklet';
const objects = objectDetectorPlugin(frame);
const boundingBox = objects[0]?.boundingBox ?? [0, 0, 0, 0];
objectBounds.value = boundingBox;
console.log(objects[0]);
console.log(objectBounds.value);
},
[objectBounds],
); if you set frameProcessorFps higher (like 30) you can get the crash happening much faster. The Java imports I am not 100% sure of as Android Studio doesn't seem to handle React Native project linting properly. |
Let me know if #351 fixes this issue |
Sadly it won't run with changes:
|
That looks like an unrelated build issue. Try cleaning your cache and try again: |
Is my goto before doing anything. |
I actually tried reverting the changes and redoing. Reverting works, but after changes won't build. |
huh, that's really weird and still seems unrelated to me, could you try rebooting your pc? |
whops merged the wrong PR.... |
It seems that this issue was sadly not fixed with #379. It crashes with object detector within 20 seconds:
Also if frameProcessorFps is lower than 4 then nothing gets detected. Also what is curious is that I sometimes get bounding box values that should not happen:
|
@Kypsis what flags are you using to collect this log? |
|
:( |
for the bounding boxes value that should not happen, try upgrading REA to the latest alpha, I believe 2.2.0 doesn't have an important PR from me yet |
@Kypsis there is a big chance of solve this issue upgrading reanimated to about the flag question sorry for my english :( what I noticed in your log is that it only contains the filters of folowing types and if I'm not mistaken when app crashes your log may include could you try |
Edit: Actually ignore the below comment for a while while I test some more. It seems simplifying the data structure sent back from Java to just an array works without crashing. With latest
|
again, this is just a stacktrace - the actual error message is missing. please check your logcat filters |
I think this issue can be closed again as resolved. Upon further testing it seems the crashes stem from Reanimated Edit: It seems I cheered to soon (running with
|
Seems returning array of arrays with only bounding boxes does not crash (tested with frameProcessor running for over 30 minutes). |
Maybe it's a race condition? I'm just so confused by the error - I'm not too familiar with native code debugging on Android, but I'd love to have a stacktrace to find out what code from VisionCamera or at least Reanimated is causing this crash, do you maybe know how you can get a full stacktrace? Maybe you have to extract C++ symbols to demystify the tombstone? We shouldn't exclude the possibility that this crash might also happen because of fbjni or Reanimated - we'll only know for sure when I have a full stacktrace... (for example in Xcode when the app crashes, the IDE pauses at the line that causes the crash, and on the left side you have a full view of all functions and line numbers that are currently in the stack, don't know how you can get that info in Android Studio) |
This keeps getting weirder (at least for me as non-native developer). Using |
With |
Could you create an issue at Reanimated? |
I'm getting this
|
Downgrading reanimated to 2.2.2 from the 2.3 alpha version seems to have resolved my issue, see software-mansion/react-native-reanimated#2256 |
I believe that's because of Hermes. Hermes introduced a new garbage collector (Hades GC) which runs on a separate Thread and the JNI environment is not attached here. I don't think this is a VisionCamera issue, but if it is feel free to re-open. |
The C++ crash that occured after running a Frame Processor for a long time (on Hermes 0.8.x+) is finally fully fixed in #455 + Reanimated v2.3.0-beta.1. |
What were you trying to do?
I was using frameProcessor for object detection.
Reproduceable Code
No response
What happened instead?
The app crashes sooner or later dependent on how high the frameProcessorFps setting is. This also occurs with imageLabeller and QRScanner plugins/solutions, just not as fast. The higher the setting the faster it crashes. Also a weird side effect is that if the frameProcessorFps is lower than 4 for object detection it doesn't detect anything, higher than 5 and you never get any labels although you will get the boundingBox. With 4 you get labels sometimes. Edit: Ignore the not finding labels correlated to frameProcessorFps, MLKit object detector classifier just has so few categories that you have to point the camera at very specific things.
Relevant log output
Device
Pixel 4a
VisionCamera Version
2.5.0
Additional information
The text was updated successfully, but these errors were encountered: