Skip to content

Commit d2c856e

Browse files
BalazsSzekeresBalazsSzekeres
committed
Added version 1.0.2 of the devkit, which now includes evaluation
1 parent 8aff90d commit d2c856e

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

82 files changed

+2190
-20462
lines changed

‎1_frame_information.ipynb

+6-2
Original file line numberDiff line numberDiff line change
@@ -451,7 +451,11 @@
451451
{
452452
"cell_type": "code",
453453
"execution_count": null,
454-
"metadata": {},
454+
"metadata": {
455+
"pycharm": {
456+
"name": "#%%\n"
457+
}
458+
},
455459
"outputs": [],
456460
"source": []
457461
}
@@ -477,4 +481,4 @@
477481
},
478482
"nbformat": 4,
479483
"nbformat_minor": 1
480-
}
484+
}

‎3_2d_visualization.ipynb

+13-4
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,11 @@
135135
},
136136
{
137137
"cell_type": "markdown",
138-
"metadata": {},
138+
"metadata": {
139+
"pycharm": {
140+
"name": "#%% md\n"
141+
}
142+
},
139143
"source": [
140144
"<a id='2d_radar'></a>\n",
141145
"## Radar point-cloud visualization\n",
@@ -145,7 +149,11 @@
145149
{
146150
"cell_type": "code",
147151
"execution_count": 4,
148-
"metadata": {},
152+
"metadata": {
153+
"pycharm": {
154+
"name": "#%%\n"
155+
}
156+
},
149157
"outputs": [
150158
{
151159
"data": {
@@ -182,7 +190,8 @@
182190
"execution_count": 5,
183191
"metadata": {
184192
"pycharm": {
185-
"is_executing": true
193+
"is_executing": true,
194+
"name": "#%%\n"
186195
}
187196
},
188197
"outputs": [
@@ -270,4 +279,4 @@
270279
},
271280
"nbformat": 4,
272281
"nbformat_minor": 1
273-
}
282+
}

‎4_evaluation.ipynb

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {
6+
"pycharm": {
7+
"name": "#%% md\n"
8+
}
9+
},
10+
"source": [
11+
"# Evaluation Example\n",
12+
"\n",
13+
"This example provides an insight on how to evaluate a model on the dataset. The same evaluation procedure is the basis of the upcoming leaderboard.\n",
14+
"For the evaluation the 'vod.evaluation' module is used. The evaluation module provides a number of metrics that can be used to evaluate a model. The metrics are:\n",
15+
"- Per class AP for the entire annotated area\n",
16+
"- Per class AP for the driving corridor\n",
17+
"- Per clas AOS for the entire annotated area\n",
18+
"- Per class AOS for the driving corridor\n",
19+
"\n",
20+
"The evaluation procedure can be used as follows:"
21+
]
22+
},
23+
{
24+
"cell_type": "code",
25+
"execution_count": 1,
26+
"metadata": {
27+
"pycharm": {
28+
"name": "#%%\n"
29+
}
30+
},
31+
"outputs": [
32+
{
33+
"name": "stdout",
34+
"output_type": "stream",
35+
"text": [
36+
"Evaluating kitti by default\n",
37+
"mAP Image BBox finished\n",
38+
"mAP bev BBox finished\n",
39+
"mAP 3D BBox finished\n",
40+
"Evaluating kitti by ROI\n",
41+
"mAP Image BBox finished\n",
42+
"mAP bev BBox finished\n",
43+
"mAP 3D BBox finished\n",
44+
"Results: \n",
45+
"Entire annotated area: \n",
46+
"Car: 25.525719831769656 \n",
47+
"Pedestrian: 25.061868496971517 \n",
48+
"Cyclist: 40.682414698162724 \n",
49+
"mAP: 30.423334342301303 \n",
50+
"Driving corridor area: \n",
51+
"Car: 89.62930867728619 \n",
52+
"Pedestrian: 68.55636970921907 \n",
53+
"Cyclist: 83.24705620654387 \n",
54+
"mAP: 80.47757819768304 \n",
55+
"\n"
56+
]
57+
}
58+
],
59+
"source": [
60+
"from vod.evaluation import Evaluation\n",
61+
"import os\n",
62+
"\n",
63+
"# When the instance is created, the label locations are required.\n",
64+
"evaluation = Evaluation(test_annotation_file=os.path.join('example_set', 'label'))\n",
65+
"\n",
66+
"# Using the evaluate method, the model can be evaluated on the detection labels.\n",
67+
"results = evaluation.evaluate(\n",
68+
" result_path=os.path.join('example_set', 'detection'),\n",
69+
" current_class=[0, 1, 2])\n",
70+
"\n",
71+
"print(\"Results: \\n\"\n",
72+
" f\"Entire annotated area: \\n\"\n",
73+
" f\"Car: {results['entire_area']['Car_3d_all']} \\n\"\n",
74+
" f\"Pedestrian: {results['entire_area']['Pedestrian_3d_all']} \\n\"\n",
75+
" f\"Cyclist: {results['entire_area']['Cyclist_3d_all']} \\n\"\n",
76+
" f\"mAP: {(results['entire_area']['Car_3d_all'] + results['entire_area']['Pedestrian_3d_all'] + results['entire_area']['Cyclist_3d_all']) / 3} \\n\"\n",
77+
" f\"Driving corridor area: \\n\"\n",
78+
" f\"Car: {results['roi']['Car_3d_all']} \\n\"\n",
79+
" f\"Pedestrian: {results['roi']['Pedestrian_3d_all']} \\n\"\n",
80+
" f\"Cyclist: {results['roi']['Cyclist_3d_all']} \\n\"\n",
81+
" f\"mAP: {(results['roi']['Car_3d_all'] + results['roi']['Pedestrian_3d_all'] + results['roi']['Cyclist_3d_all']) / 3} \\n\"\n",
82+
" )"
83+
]
84+
},
85+
{
86+
"cell_type": "code",
87+
"execution_count": null,
88+
"metadata": {},
89+
"outputs": [],
90+
"source": []
91+
}
92+
],
93+
"metadata": {
94+
"kernelspec": {
95+
"display_name": "Python 3",
96+
"language": "python",
97+
"name": "python3"
98+
},
99+
"language_info": {
100+
"codemirror_mode": {
101+
"name": "ipython",
102+
"version": 3
103+
},
104+
"file_extension": ".py",
105+
"mimetype": "text/x-python",
106+
"name": "python",
107+
"nbconvert_exporter": "python",
108+
"pygments_lexer": "ipython3",
109+
"version": "3.7.12"
110+
}
111+
},
112+
"nbformat": 4,
113+
"nbformat_minor": 1
114+
}

‎CHANGELOG.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
<!--next-version-placeholder-->
2+
3+
## v1.0.0 (2022-07-01)
4+
### Feature
5+
* **vod/evaluation:** New feature added that evaluates the detections from the vod dataset ([`18ae1b1`](https://gitlab.tudelft.nl/intelligent-vehicles/view-of-delft-dataset/-/commit/18ae1b18d2dd29b6d8006d560b9687923306b709))
6+
* **vod.common:** A package for common functions ([`868cd08`](https://gitlab.tudelft.nl/intelligent-vehicles/view-of-delft-dataset/-/commit/868cd08288ff4b8d40f294e29d1c047a0dc8d141))
7+
* **vod.evaluation:** Added evaulate package structure based on evalai ([`9a3e06f`](https://gitlab.tudelft.nl/intelligent-vehicles/view-of-delft-dataset/-/commit/9a3e06f961b7f1e1eca695f0b564b8b650d9a68f))
8+
9+
### Breaking
10+
* ([`18ae1b1`](https://gitlab.tudelft.nl/intelligent-vehicles/view-of-delft-dataset/-/commit/18ae1b18d2dd29b6d8006d560b9687923306b709))

‎README.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -102,7 +102,7 @@ In case of questions of problems, please send an email to a.palffy at tudelft.nl
102102

103103
## Getting Started
104104

105-
Please refer to the [GETTING_STARTED](docs/GETTING_STARTED.md) manual to learn how to organize the data and start using the development kit.
105+
Please refer to the [GETTING_STARTED](docs/GETTING_STARTED.md) manual to learn how to organize the data and start using the development kit, as well as find information regarding evaluation.
106106
<br>
107107
<br>
108108

‎common/__init__.py

-1
This file was deleted.

‎common/file_handling.py

-11
This file was deleted.

‎docs/GETTING_STARTED.md

+18-1
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ All the codes are tested in the following environment:
4141
* Python 3.6+
4242
* Note that Python 3.6+ is required for the devkit and its examples, but the dataset can be accessed with any programming language and Python version.
4343

44-
### Installation of the Devkit
44+
### Installation of the Devkit from GitHub
4545

4646
1. Clone the repository: `git@github.com:tudelft-iv/view-of-delft-dataset.git`
4747
2. Inside the root folder, use the `environment.yml` to create a new conda environment using: `conda env create -f environment.yml`
@@ -50,10 +50,27 @@ All the codes are tested in the following environment:
5050

5151
In case the interactive plots do not show up in the notebooks use: `jupyter nbextension install --py --sys-prefix k3d`
5252

53+
### Installation of the Devkit using package manager
54+
The devkit is also available as a pip package. To install the devkit, run the following command:
55+
```
56+
pip install vod-tudelft
57+
```
58+
5359
### Usage
5460
After fetching both the data and the devkit, please refer to these manuals for several examples of how to use them, including data loading, fetching and applying transformations, and 2D/3D visualization:
5561
[Frame Information](https://github.com/tudelft-iv/view-of-delft-dataset/blob/main/1_frame_information.ipynb)
5662
[Frame Transformations](https://github.com/tudelft-iv/view-of-delft-dataset/blob/main/2_frame_transformations.ipynb)
5763
[2D Visualization](https://github.com/tudelft-iv/view-of-delft-dataset/blob/main/3_2d_visualization.ipynb)
5864

65+
### Evaluation
66+
The devkit provides a set of evaluation scripts that can be used to evaluate the performance of your model.
67+
The evaluation script is located under `vod.evaluation`. The fourth notebook provides an example of how to use the evaluation script.
68+
69+
The evaluation script is based on the origin Kitti C++ code, which has been translated to Python by a Github user
70+
named [Yan Yan](https://github.com/traveller59/kitti-object-eval-python). Our contribution to the evaluation script are:
71+
- Removing CUDA dependency
72+
- Adding further evaluation metrics
73+
- Fixing Numba issues
74+
- PEP8 style guide
75+
5976

‎docs/api_docs/.buildinfo

-4
This file was deleted.

‎docs/api_docs/_sources/index.rst.txt

-20
This file was deleted.

‎docs/api_docs/_sources/modules.rst.txt

-7
This file was deleted.

‎docs/api_docs/_sources/vod.configuration.rst.txt

-21
This file was deleted.

‎docs/api_docs/_sources/vod.frame.rst.txt

-37
This file was deleted.

‎docs/api_docs/_sources/vod.rst.txt

-20
This file was deleted.

0 commit comments

Comments
 (0)