Object Detection #1930

warthecatalyst · 2022-08-19T08:14:44Z

This PR is mainly about yolov3 model in ObjectDetection.

add Yolov3 Model in basic model zoo and can be trained with a simple loss function for Pytorch engine. And corresponding test.
add Yolov3 Loss function, maybe still have some bugs to fix
add an example of yolov3Training: TrainPikachuWithYOLOV3
About ObjectDetectionDataSet, add a new function getClasses() because for most ObjectDetection Models and Translators, we need to know how many types of objects in the dataset.
Add ObjectDetection for DJL-Zero, and due to small bugs for yolov3, currently just train a SingleShotDetection Model.

TODO List:

Translator of Yolov3(actually I'm written a demo but it can't pass the code check so I stash it.
For COCODataSet, we should update synset.

api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java

model-zoo/src/main/java/ai/djl/basicmodelzoo/cv/object_detection/yolo/YOLOV3.java

frankfliu · 2022-08-29T05:06:08Z

api/src/main/java/ai/djl/training/loss/SingleShotDetectionLoss.java

@@ -48,6 +48,8 @@ public SingleShotDetectionLoss() {
    @Override
    protected Pair<NDList, NDList> inputForComponent(
            int componentIndex, NDList labels, NDList predictions) {
+        System.out.println(labels.singletonOrThrow());  //print labels for test


Remove testing code

api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java

frankfliu · 2022-08-29T05:06:45Z

api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java

+import java.util.ArrayList;
+import java.util.Arrays;
+
+public class YOLOv3Loss extends Loss{


./gradlew formatJava

frankfliu · 2022-08-29T05:46:26Z

api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java

+        return inter.div(union);
+    }
+
+    public ArrayList<PairList<Long,Rectangle>> getTargetFromCurrentLabel(NDArray labels){


Use List instead of ArrayList in function declaration

frankfliu · 2022-08-29T05:46:54Z

api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java

+    }
+
+    //calculate IOU is already defined in Rectangle
+    public NDArray calculateIOU(NDArray boxA,NDArray boxB){


frankfliu · 2022-08-29T05:48:18Z

examples/src/main/java/ai/djl/examples/training/TrainPikachu.java

@@ -87,7 +87,7 @@ public static TrainingResult runExample(String[] args) throws IOException, Trans
            try (Trainer trainer = model.newTrainer(config)) {
                trainer.setMetrics(new Metrics());

-                Shape inputShape = new Shape(arguments.getBatchSize(), 3, 256, 256);
+                Shape inputShape = new Shape(1, 3, 256, 256);


You probably shouldn't modify the example to use a fixed batch size of 1 rather than the variable version

warthecatalyst · 2022-08-29T05:56:51Z

emm,this is just a test pr for my mentor Zack to get a view on my current progress and make suggestions At 2022-08-29 13:50:35, "Frank Liu" ***@***.***> wrote: @frankfliu commented on this pull request. In api/src/main/java/ai/djl/training/loss/SingleShotDetectionLoss.java:

@@ -48,6 +48,8 @@ public SingleShotDetectionLoss() {

@OverRide protected Pair<NDList, NDList> inputForComponent( int componentIndex, NDList labels, NDList predictions) { + System.out.println(labels.singletonOrThrow()); //print labels for test Remove testing code In api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java:

@@ -0,0 +1,275 @@

+package ai.djl.training.loss; Add license header In api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java:

+import ai.djl.modality.cv.output.Rectangle;

+import ai.djl.ndarray.NDArray; +import ai.djl.ndarray.NDArrays; +import ai.djl.ndarray.NDList; +import ai.djl.ndarray.NDManager; +import ai.djl.ndarray.index.NDIndex; +import ai.djl.ndarray.types.DataType; +import ai.djl.ndarray.types.Shape; +import ai.djl.nn.Activation; +import ai.djl.util.Pair; +import ai.djl.util.PairList; + +import java.util.ArrayList; +import java.util.Arrays; + +public class YOLOv3Loss extends Loss{ ./gradlew formatJava In api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java:

+ boxBTrue.get(":,2:").expandDims(0).broadcast(A,B,2));

+ NDArray minXY = NDArrays.maximum(boxATrue.get(":,:2").expandDims(1).broadcast(A,B,2), + boxBTrue.get(":,:2").expandDims(0).broadcast(A,B,2)); + + NDArray inter = NDArrays.minimum(maxXY.sub(minXY),0); + inter = inter.get(":,:,0").mul(inter.get(":,:,1")); + + //to calculate the area of prediction bbox and true bbox + NDArray areaA = boxATrue.get(":,2").sub(boxATrue.get(":,0")).mul(boxATrue.get(":,3").sub(boxATrue.get(":,1"))).expandDims( 1).broadcast(inter.getShape()), + areaB = boxBTrue.get(":,2").sub(boxBTrue.get(":,0")).mul(boxBTrue.get(":,3").sub(boxBTrue.get(":,1"))).expandDims( 0).broadcast(inter.getShape()); + + NDArray union = areaA.add(areaB).sub(inter); + return inter.div(union); + } + + public ArrayList<PairList<Long,Rectangle>> getTargetFromCurrentLabel(NDArray labels){ Use List instead of ArrayList in function declaration In api/src/main/java/ai/djl/training/loss/YOLOv3Loss.java:

+ float interHeight = Math.min(trueBottom,predBottom)-Math.max(trueTop,predTop);

+ float inter = interWidth*interHeight, union = wgt.get(i).getFloat()*inW*hgt.get(i).getFloat()*inH + + anchors.get(j,0).getFloat()*anchors.get(j,1).getFloat()-inter; + iou.set(new NDIndex(curIndex),inter/union); + } + } + return new NDList(iou,boxLossScale,groundTruth); + } + + + + return null; + } + + //calculate IOU is already defined in Rectangle + public NDArray calculateIOU(NDArray boxA,NDArray boxB){ javadoc In examples/src/main/java/ai/djl/examples/training/TrainPikachu.java:

@@ -87,7 +87,7 @@ public static TrainingResult runExample(String[] args) throws IOException, Trans

try (Trainer trainer = model.newTrainer(config)) { trainer.setMetrics(new Metrics()); - Shape inputShape = new Shape(arguments.getBatchSize(), 3, 256, 256); + Shape inputShape = new Shape(1, 3, 256, 256); Why? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

zachgk · 2022-10-06T23:26:25Z

basicdataset/src/main/java/ai/djl/basicdataset/cv/CocoDetection.java

+    /** {@inheritDoc} */
+    @Override
+    public List<String> getClasses() {
+        return synset;


It is better to fail clearly rather than have something partial that may confuse people that it works. Here, just throw an exception, specifically at UnsupportedOperationException to indicate that it is not yet implemented.

zachgk · 2022-10-06T23:27:49Z

examples/src/main/java/ai/djl/examples/training/TrainPikachu.java

@@ -87,7 +87,7 @@ public static TrainingResult runExample(String[] args) throws IOException, Trans
            try (Trainer trainer = model.newTrainer(config)) {
                trainer.setMetrics(new Metrics());

-                Shape inputShape = new Shape(arguments.getBatchSize(), 3, 256, 256);
+                Shape inputShape = new Shape(1, 3, 256, 256);


You probably shouldn't modify the example to use a fixed batch size of 1 rather than the variable version

zachgk · 2022-10-06T23:28:51Z

integration/src/main/java/ai/djl/integration/tests/model_zoo/object_detection/YOLOv3Test.java

+
+public class YOLOv3Test {
+    @Test
+    public void testDarkNet() {


You should add a test for the full yolov3 model as well

1. add Model of Yolov3(completely right I think, at least for Pytorch engine) 2. add Loss function of Yolov3 (still have some small bugs, it can be successfully trained for 4-5 minutes)... 3. add ObjectDetection in djl-zero, and currently just training SingleShotDetection Model 4. For ObjectDetectionDataSet, add a new method getClasses(), because for most models and Translators, we still need the classes of dataset

warthecatalyst requested review from zachgk and frankfliu as code owners August 19, 2022 08:14

zachgk marked this pull request as draft August 19, 2022 16:56

zachgk reviewed Aug 26, 2022

View reviewed changes

warthecatalyst force-pushed the ObjectDetection branch 2 times, most recently from 3b530c1 to 7474f50 Compare August 26, 2022 01:12

frankfliu reviewed Aug 29, 2022

View reviewed changes

warthecatalyst force-pushed the ObjectDetection branch 2 times, most recently from afa3241 to efcaa88 Compare October 5, 2022 15:31

warthecatalyst marked this pull request as ready for review October 6, 2022 06:18

warthecatalyst changed the title ~~Object detection~~ Object Detection Oct 6, 2022

zachgk requested changes Oct 6, 2022

View reviewed changes

warthecatalyst added 2 commits October 7, 2022 10:02

fix some bugs of ObjectDetection and add testofYolov3

79900ea

warthecatalyst force-pushed the ObjectDetection branch from efcaa88 to 79900ea Compare October 7, 2022 02:02

zachgk approved these changes Oct 7, 2022

View reviewed changes

zachgk merged commit b25ec9c into deepjavalibrary:master Oct 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Object Detection #1930

Object Detection #1930

warthecatalyst commented Aug 19, 2022 •

edited

Loading

frankfliu Aug 29, 2022

frankfliu Aug 29, 2022

frankfliu Aug 29, 2022

frankfliu Aug 29, 2022

frankfliu Aug 29, 2022

zachgk Oct 6, 2022

warthecatalyst commented Aug 29, 2022 via email

zachgk Oct 6, 2022

zachgk Oct 6, 2022

zachgk Oct 6, 2022

Object Detection #1930

Object Detection #1930

Conversation

warthecatalyst commented Aug 19, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

warthecatalyst commented Aug 29, 2022 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

warthecatalyst commented Aug 19, 2022 •

edited

Loading