New model parsers. #5

kkeroo · 2024-06-21T13:20:24Z

New host nodes

Xfeat model: to get the matching points, 2 images are needed, so on first iteration the results are stored and then next frame is matched with the initial frame. Not sure how we want to define reference and target frames in the entire pipeline? So, the parser assumes that first frame that comes to the NN is the reference frame. In practice, the first frame is probably not the best idea and we should set the reference frame over some set function?
SuperAnimal landmark model
M-LSD line detector: some more linking is used. parser node accepts two inputs: one is the output from the NN and the other is the passthrough from the NN. The latter is needed for postprocessing the results.
Mediapipe Face mesh model

Messages and utils

XFeat - MatchedPoints custom message and function for creating the message added. Util functions for decoding Xfeat results are added in separate file.
SuperAnimal - creating message with generic Keypoints message class. Also, util functions for decoding the results.
FaceMesh - using the same create message function as SuperAnimal (making it generic)
M-LSD - Lines and Line custom messages and function for creating the message added. Also, util functions for decoding the results.

Additionally, some parts are adapted to the #4 from @jkbmrz.

ml/postprocessing/utils/xfeat.py

jkbmrz · 2024-06-24T08:49:08Z

ml/postprocessing/utils/xfeat.py

+                            (0, 1, 3, 2, 4)).reshape(B, 1, H * 8, W * 8)
+    return heatmap
+
+def _nms(


We use NMS many times in this repo. Should we write a single general NMS function that would be used throughout the library?

This would be great. Maybe address this in a new PR?

Agree. We can make some parsing PR where we fix such things.

I also think the above bilinear grid sample is generic enough to be in some common utils rather than specific to xfeat. We should try and make them as generic as possible, so they can be easily re-used by other nodes where it makes sense. Let's address those in a separate PR.

ml/postprocessing/utils/xfeat.py

jkbmrz · 2024-06-24T09:04:55Z

ml/messages/creation_functions/detection.py

@@ -113,3 +115,51 @@ def create_detection_message(
    detections_msg = img_detections()
    detections_msg.detections = detections
    return detections_msg
+
+def create_line_detection_message(lines: np.ndarray, scores: np.ndarray):


Should we merge this with the more general create_detection_message function?

Hmm, I dont think we should. First, meaning of the two functions is not the same, e.g. create_detection_function is used for creating messages with bboxes and create_line_detection_message is used for creating messages with lines. Second, functions use different message types, bbox detections have xmin, xmax values but Line message does not really have min and max coordinates as line can be pointed in any direction. Third, validation will be more complex.

Overall, I think we would only complicate things to much.

@jkbmrz thoughts?

Okay I see. At first thought it made sense to merge because create_detection_message already includes various types of "detections" and not all of them are obligatory (e.g. keypoints). But as there is indeed no situation where lines would be detected together with bboxes so let's keep them separated. I'd maybe move the create_line_detection_message into a new file named as lines.py to avoid the ambiguity (it has nothing to do with object detection). Moreover, to avoid further ambiguity, we can think of renaming detection.py (e.g. to object_detection.py or objects.py) and rename it's message to create_object_detection_message (but we can do this in another PR).

Let's move this to lines.py I agree. We can leave detection.py as is given it's typically associated with object detection.

tersekmatija · 2024-06-24T12:04:26Z

Let's first rebase based on #4 , then we can go ahead and review as it seems there are duplicate files.

jkbmrz · 2024-06-24T13:11:22Z

Let's first rebase based on #4 , then we can go ahead and review as it seems there are duplicate files.

#4 is merged.

tersekmatija

Decent base, requested some minor modifications. We can merge some utils functions in later PRs.

ml/messages/creators/keypoints.py

tersekmatija · 2024-06-24T14:51:29Z

ml/messages/creators/matched_points.py

I think we could instead do something like: https://docs.luxonis.com/software/depthai-components/nodes/feature_tracker.

We would likely be matching features across the time, so perhaps using MatchedFeatures from above would make the most sense?

Alternative option is to provide 2 possible decoders, one for feature matching and one for feature tracking.

Okay, meaning that I add TrackedFeature in the list of reference_points and target_points inside MatchedPoints or full support with TrackedFeatures msg consisting of TrackedFeature? If the latter, how can I "connect" (match) two features? By setting the same id and increasing age?

Yes, you would need to set the same ID to the new point and increment the age. I would do a full support of TrackedFeatures so that DAI can use this directly or see how it can be used downstream.

Let's clarify with DAI team internally if we have some further questions on this.

Added in e50e75a.

ml/postprocessing/mediapipe_face_landmarker.py

tersekmatija · 2024-06-24T14:54:44Z

ml/postprocessing/mediapipe_face_landmarker.py

+
+from ..messages.creators import create_keypoints_message
+
+class MPFaceLandmarkerParser(dai.node.ThreadedHostNode):


Could this be more generic? Something like a KeypointParser or something along those lines indicating that it's a Parser for keypoints output on a single image?

Not sure if it can be generic enough so we can for example join it with SuperAnimal Landmarker?

General keypoint parser added in 2eefa6e.

Note: face_landmark blob returns [1,1,1,1404] so simple calcuation is performed to get the num. of coords. (2 or 3).

I think SuperAnimal cannot be merged because it is pruned network and requires some additional postprocessing.

Note: face_landmark blob returns [1,1,1,1404] so simple calcuation is performed to get the num. of coords. (2 or 3).

I would assume that at times we could have outputs of [B, N_keypoints, D_keypoints] where N is the number and D dimension. Potentially, we have support for this and option for reshape like in case of face_landmark output?

Yes. We only assume that the batch size is 1.

ml/postprocessing/utils/mlsd.py

ml/postprocessing/utils/superanimal.py

tersekmatija · 2024-06-24T14:58:51Z

ml/postprocessing/utils/xfeat.py

+                            (0, 1, 3, 2, 4)).reshape(B, 1, H * 8, W * 8)
+    return heatmap
+
+def _nms(


I also think the above bilinear grid sample is generic enough to be in some common utils rather than specific to xfeat. We should try and make them as generic as possible, so they can be easily re-used by other nodes where it makes sense. Let's address those in a separate PR.

ml/postprocessing/keypoint.py

tersekmatija

Minor changes request but fine to merge afterwards.

tersekmatija · 2024-06-27T11:21:25Z

ml/messages/creators/keypoints.py

+        Keypoints: Message containing 2D or 3D coordinates of the detected keypoints.
+    """
+


Note is that his might require reshaping depending on the output as mentioned in one of the comments.

Reshape is handled in the parser.

tersekmatija · 2024-06-27T11:24:09Z

ml/postprocessing/mediapipe_hand_landmarker.py

I'm thinking if this could be somehow fused with likes of YuNet or Keypoints instead.

Effectively, this will output keypoints + two scores for the image. We can explore this later as well potentially, but I think worth bringing up. Thoughts?

Hmm, yea probably with Keypoints as its closer in meaning. Maybe we can extend Keypoints so that it expects more than one output tensor. We would assume that the first tensor will represent keypoints, and other additional tensors will represent some scores? Other keypoint models will maybe also have score that the object is present in the picture or something.

However not sure if this is the general case with all keypoint models (to have additional scalar outputs).

kkeroo requested review from jkbmrz and tersekmatija and removed request for jkbmrz and tersekmatija June 21, 2024 13:38

jkbmrz requested changes Jun 24, 2024

View reviewed changes

kkeroo requested a review from jkbmrz June 24, 2024 10:48

tersekmatija mentioned this pull request Jun 24, 2024

Restructure; Add Parsers, Util Functions and Message Creators #4

Merged

kkeroo force-pushed the feature/add_models branch from 36701a7 to 5e54ee4 Compare June 24, 2024 12:25

kkeroo added 7 commits June 24, 2024 15:18

DLC Superanimal parser added.

42710ad

Add superanimal to init.

065c480

Parser for XFeat model added.

5ea5e8d

MediaPipe face mesh parser added with general keypoint msg. creation.

9b2c230

M-LSD parser added.

148dc05

Variables renaming.

495ab3f

Layout check.

6e7a2c4

kkeroo force-pushed the feature/add_models branch from 5e54ee4 to 6e7a2c4 Compare June 24, 2024 13:23

Import fixes.

920565c

tersekmatija requested changes Jun 24, 2024

View reviewed changes

kkeroo self-assigned this Jun 24, 2024

kkeroo added 2 commits June 26, 2024 12:00

List support for keypoints.

bb90cfa

Remove dequantization.

2b33133

kkeroo added 2 commits June 26, 2024 15:56

General keypoint parser.

2eefa6e

Rename.

f211aad

kkeroo requested a review from tersekmatija June 26, 2024 14:02

jkbmrz requested changes Jun 26, 2024

View reviewed changes

ml/postprocessing/keypoint.py Outdated Show resolved Hide resolved

Default values.

48f280c

kkeroo requested a review from jkbmrz June 26, 2024 16:06

jkbmrz approved these changes Jun 27, 2024

View reviewed changes

tersekmatija approved these changes Jun 27, 2024

View reviewed changes

MatchedPoint replaced with dai.TrackedFeatures.

e50e75a

kkeroo merged commit 0c0edd6 into main Jun 28, 2024

kkeroo deleted the feature/add_models branch June 28, 2024 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New model parsers. #5

New model parsers. #5

kkeroo commented Jun 21, 2024 •

edited

Loading

jkbmrz Jun 24, 2024

kkeroo Jun 24, 2024

jkbmrz Jun 24, 2024

tersekmatija Jun 24, 2024

jkbmrz Jun 24, 2024

kkeroo Jun 24, 2024

jkbmrz Jun 24, 2024

tersekmatija Jun 27, 2024

tersekmatija commented Jun 24, 2024

jkbmrz commented Jun 24, 2024

tersekmatija left a comment

tersekmatija Jun 24, 2024

kkeroo Jun 26, 2024

tersekmatija Jun 27, 2024

kkeroo Jun 28, 2024

tersekmatija Jun 24, 2024

tersekmatija Jun 24, 2024

kkeroo Jun 26, 2024

tersekmatija Jun 27, 2024

kkeroo Jun 28, 2024

tersekmatija Jun 24, 2024

tersekmatija left a comment

tersekmatija Jun 27, 2024

kkeroo Jun 28, 2024

tersekmatija Jun 27, 2024

kkeroo Jun 28, 2024 •

edited

Loading


		from ..messages.creators import create_keypoints_message

		class MPFaceLandmarkerParser(dai.node.ThreadedHostNode):

		Keypoints: Message containing 2D or 3D coordinates of the detected keypoints.
		"""

New model parsers. #5

New model parsers. #5

Conversation

kkeroo commented Jun 21, 2024 • edited Loading

New host nodes

Messages and utils

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tersekmatija commented Jun 24, 2024

jkbmrz commented Jun 24, 2024

tersekmatija left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tersekmatija left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkeroo Jun 28, 2024 • edited Loading

Choose a reason for hiding this comment

kkeroo commented Jun 21, 2024 •

edited

Loading

kkeroo Jun 28, 2024 •

edited

Loading