Mp Hands nodes & segmentation msg creation #2

kkeroo · 2024-05-28T08:00:48Z

Segmentation msg creation

Segmentation message is created from dai.ImgFrame with RAW8 type because we only have a few classes. @jkbmrz Can you add this message for depth model using RAW16 type?

MediaPipe Hands nodes

Two nodes for two models are added: Palm detection and Hand Landmark. To generate anchors and decode detections mediapipe_utils.py is added. Palm detector outputs classical detection message, Landmark model outputs custom HandLandmark message, consisting of confidence, handdedness and landmarks. Handdedness is just info about left or right hand. In the future when new pose detection models are added, we will maybe have more general msg containing only confidence and landmarks and then extend this class for the hands,... And the landmarks is just list of dai.Point3f

jkbmrz · 2024-05-28T09:47:36Z

Can you add this message for depth model using RAW16 type?

Will add later today!

ml/messages/landmarks.py

tersekmatija · 2024-05-28T09:11:01Z

ml/messages/landmarks.py

+        dai.Buffer.__init__(self)
+        self.confidence: float = 0.0
+        self.handdedness: float = 0.0
+        self.landmarks: List[dai.Point3f] = []


We might want to use descriptors to validate that landmarks are of correct class?

@kkeroo I've just added a PR (#3) with descriptors, if that comes handy!

Added @property and @property.setter to match PR #3.

tersekmatija · 2024-05-28T09:14:30Z

ml/postprocessing/mp_hand_landmark.py

+        self,
+        score_threshold=0.5,
+        handdedness_threshold=0.5,
+        input_size=(224, 224)


It doesn't seem this is being used anywhere?

Also, a note here is that we might want to return normalized 0-1 values. DepthAI does the same for the bounding boxes. And there is some beauty in it, where you don't need to know the shape in advance to unnormalize.

We can then have a "visualize" method on the Parser or Message itself, or a static method in the parser, that can visualize the message onto a specific frame that is passed together. That way we can nicely avoid this.

Do we want to do this on for all nodes? Right now, face detector parser node (scrfd) returns bboxes w.r.t. input size. Should I also normalize there?

tersekmatija · 2024-05-28T09:14:58Z

ml/postprocessing/mp_hand_landmark.py

+
+    def run(self):
+        """
+        Postprocessing logic for SCRFD model.


This must be improved.

Added better description here

tersekmatija · 2024-05-28T09:16:45Z

ml/postprocessing/mp_hand_landmark.py

+            print(f"Layer names = {output.getAllLayerNames()}")
+
+            tensorInfo = output.getTensorInfo("Identity")
+            landmarks = output.getTensor(f"Identity").reshape(21, 3).astype(np.float32)


Might be worth checking if dequantization can be performed by DepthAI itself. Something like
output.getTensor("name", dequantize=True) where True is the default value could be beneficial. Please open a new thread on Slack to request support.

Opened a thread.

tersekmatija · 2024-05-28T09:24:28Z

ml/postprocessing/selfie_seg.py

mediapipe_selfie_segmentation

tersekmatija · 2024-05-28T09:24:43Z

ml/postprocessing/selfie_seg.py

@@ -1,24 +1,20 @@
 import depthai as dai
 import numpy as np
 import cv2
+from .utils.message_creation.depth_segmentation import create_depth_segmentation_msg

 class SeflieSegParser(dai.node.ThreadedHostNode):
    def __init__(
        self,
        threshold=0.5,
        input_size=(256, 144),


I'd eliminate this as well if we can.

So, no general function for creating segmentation messages? This step will repeat in other seg. parsers as well. It is also very similar to depth message.

Then I should add message creation back to the parser?

Okay, I see that bellow is the proposal for creating specific msg.

Yeah we can either do a specific message if that works well for DAI, or we define such models for all types of outputs to not increase message complexities where it's not needed?

Implemented function to create messages here

tersekmatija · 2024-05-28T09:25:09Z

ml/postprocessing/utils/medipipe_utils.py

mediapipe.py

Is this file taken from somewhere? If yes, we should cite the source and check whether license is permissive or restrictive?

It might make sense we only keep the methods that are useful for us, and if there aren't many, maybe we should rewrite if license is restrictive.

Are there some methods that are useful in other detection models?

tersekmatija · 2024-05-28T09:31:41Z

ml/postprocessing/utils/message_creation/depth_segmentation.py

One option would be we define this as a message itself? Not sure how visualizers or DAI would work, but if we extend dai.ImgFrame in DepthMessage without any additional fields it might be fine and we wouldn't need this. We would just better initialize self in the constructor.

In any case, I believe this would be better suited for messages utils. But in such case we would want to have create method for all types of messages, not just some, so maybe above solution avoid the need for that. Not sure if such methods would be useful for more complex messages though.

Implemented function to create messages here

tersekmatija · 2024-05-28T11:23:37Z

ml/postprocessing/utils/medipipe_utils.py

+def normalize_radians(angle):
+    return angle - 2 * math.pi * math.floor((angle + math.pi) / (2 * math.pi))
+
+def non_maxima_suppression(bboxes, iou_threshold):


Is this needed if we use cv2 NMS elsewhere?

Not needed.

jkbmrz · 2024-05-28T09:39:17Z

ml/messages/landmarks.py

+        dai.Buffer.__init__(self)
+        self.confidence: float = 0.0
+        self.handdedness: float = 0.0
+        self.landmarks: List[dai.Point3f] = []


Landmarks are the same as keypoints, right? Let's use an uniform naming (I'd prefer keypoints as this is also used in luxonis-train)

Okay, makes sense. Renamed it to keypoints.

jkbmrz · 2024-05-28T09:40:51Z

ml/postprocessing/mp_hand_landmark.py

+
+    def run(self):
+        """
+        Postprocessing logic for SCRFD model.


Is the model named mp_hand_landmark or SCFRD? If second, then naming of this file is incorrect

Added the right description to all nodes.

jkbmrz · 2024-05-28T09:41:35Z

ml/postprocessing/mp_hand_landmark.py

+                break  # Pipeline was stopped
+
+            print('MP Hand landmark node')
+            print(f"Layer names = {output.getAllLayerNames()}")


I guess these 2 print statements can be omitted

Removed all print statements.

jkbmrz · 2024-05-28T13:16:17Z

ml/messages/landmarks.py

+        dai.Buffer.__init__(self)
+        self.confidence: float = 0.0
+        self.handdedness: float = 0.0
+        self.landmarks: List[dai.Point3f] = []


@kkeroo I've just added a PR (#3) with descriptors, if that comes handy!

tersekmatija · 2024-05-31T08:38:18Z

ml/postprocessing/mediapipe_selfie_segmentation.py

+import cv2
+from .utils.message_creation import create_segmentation_message
+
+class MPSeflieSegParser(dai.node.ThreadedHostNode):


Can we make this a general segmentation parser? Same comment would apply for other classes.
In other words, is there anything super specific that this must be MPSelfieSeg or can it just be SegmentationParser?

We can define it as general segmentation parser for two classes (front, background). Actually it is already like that. Or should we extend it to multiclass? Maybe better if we have multiclass seg. parser separate?

I don't think it differs much? So I'd make it general and multiclass.

tersekmatija · 2024-05-31T12:18:18Z

ml/postprocessing/mediapipe_hand_detection.py

+            bboxes = np.array(bboxes)[indices]
+            scores = np.array(scores)[indices]
+
+            detections = []


Shouldn't this be in a create function?

Function added here.

tersekmatija · 2024-05-31T12:21:05Z

ml/postprocessing/mediapipe_hand_landmarker.py

+            # normalize landmarks
+            landmarks /= self.scale_factor
+
+            hand_landmarks_msg = HandKeypoints()


create_message function?

Function added here.

tersekmatija · 2024-05-31T12:22:22Z

ml/postprocessing/utils/detection.py

@@ -0,0 +1,11 @@
+from .medipipe import generate_handtracker_anchors, decode_bboxes, detections_to_rect, rect_transformation
+
+def generate_anchors_and_decode(bboxes, scores, threshold=0.5, scale=192):


Given this only contains mediapipe functions, I don't think it belongs here.

When we adapt mediapipe utils and make it more generic, we can move certain parts here.

tersekmatija · 2024-05-31T12:24:30Z

ml/postprocessing/segmentation.py

+            segmentation_mask = segmentation_mask[0]  # num_clases x H x W
+            overlay_image = np.zeros((segmentation_mask.shape[1], segmentation_mask.shape[2], 1), dtype=np.uint8)
+
+            for class_id in range(self.num_classes-1):


Shouldnt overlay_image be just an argmax over segmentation mask?

Yes, argmax added here. One 'layer' is added in the first dimension with all zeros so first class has index 1.

ml/postprocessing/utils/message_creation/depth.py

ml/postprocessing/utils/message_creation/segmentation.py

tersekmatija · 2024-06-06T08:31:21Z

ml/postprocessing/utils/message_creation/segmentation.py

+
+    if not isinstance(x, np.ndarray):
+        raise ValueError(f"Expected numpy array, got {type(x)}.")
+    if len(x.shape) != 3:


Should we validate there is only one channel as well?

Yes, added in f8a48f1.

tersekmatija · 2024-06-06T08:32:30Z

ml/postprocessing/utils/message_creation/depth.py

+        raise ValueError(f"Expected numpy array, got {type(x)}.")
+    if len(x.shape) != 3:
+        raise ValueError(f"Expected 3D input, got {len(x.shape)}D input.")
+


Should we validate that there is only one channel as well?

Yes, added in f8a48f1.

kkeroo added 5 commits May 28, 2024 09:30

New import path.

38be7e8

Segmentation msg creation and new selfi seg. output.

c7a566f

Nodes for MP hands.

e7dc367

More general function for depth or segmentation message creation.

eac159c

Typo.

65c8b75

kkeroo requested a review from jkbmrz May 28, 2024 08:00

tersekmatija requested changes May 28, 2024

View reviewed changes

kkeroo added 4 commits May 29, 2024 17:02

Adding HandLandmarksDescriptor.

869b607

Removing unnecessary variables.

3434ca3

Renaming mp hands nodes.

d796a21

Rename MP selfie segmentation node.

26a53e4

jkbmrz reviewed May 30, 2024

View reviewed changes

kkeroo added 5 commits May 30, 2024 15:35

HandKeypoints message using property.

5437ca8

Depth and Segmentation msg creation.

9a486bc

Segmentation msg added to host node.

c084392

Removed unnecessary prints.

b1a5ce0

Renamed mediapipe.

d7171e8

jkbmrz approved these changes May 30, 2024

View reviewed changes

Normalize hand landmarks.

df7a38b

tersekmatija reviewed May 31, 2024

View reviewed changes

kkeroo added 3 commits May 31, 2024 11:22

Anchors and decoding in utils.

8d6f6b6

Mediapipe metadata added.

3e90e4b

general multiclass segmentation parser.

bf463a2

kkeroo requested a review from tersekmatija May 31, 2024 10:42

tersekmatija requested changes May 31, 2024

View reviewed changes

kkeroo added 4 commits June 3, 2024 17:58

Validating input in create functions.

26381fa

Function for creating detection and keypoint msgs.

2deaeee

Moved anchors function.

293ce23

argmax to get segmentation classes.

45131f7

kkeroo requested a review from tersekmatija June 3, 2024 17:11

kkeroo added 5 commits June 3, 2024 20:00

Updated docs.

827d5c8

Keypoints message.

25ce9cc

Generic keypoints.

47b8dd7

Correction: handdedness -> handedness.

847165f

Init file changed.

e2d0534

tersekmatija approved these changes Jun 6, 2024

View reviewed changes

kkeroo added 3 commits June 6, 2024 11:16

1 channel in 3rd dim. validation.

f8a48f1

Remove unnecessary atributes.

14a9e4c

Merge branch 'main' into mp_hands

7d9f6d4

kkeroo merged commit 9edd524 into main Jun 6, 2024

tersekmatija mentioned this pull request Jun 6, 2024

Restructure; Add Parsers, Util Functions and Message Creators #4

Merged

kkeroo deleted the mp_hands branch July 4, 2024 07:30

		@@ -0,0 +1,11 @@
		from .medipipe import generate_handtracker_anchors, decode_bboxes, detections_to_rect, rect_transformation

		def generate_anchors_and_decode(bboxes, scores, threshold=0.5, scale=192):

Mp Hands nodes & segmentation msg creation #2

Mp Hands nodes & segmentation msg creation #2

Conversation

kkeroo commented May 28, 2024 • edited Loading

Segmentation msg creation

MediaPipe Hands nodes

jkbmrz commented May 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkeroo May 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kkeroo commented May 28, 2024 •

edited

Loading

kkeroo May 30, 2024 •

edited

Loading