🚀 MVCTrack: Boosting 3D Point Cloud Tracking 🚀

Authors: Zhaofeng Hu^1†, Sifan Zhou^2†*, Shibo Zhao³, Zhihang Yuan^4
1Stony Brook University, ²Southeast University, ³Carnegie Mellon University, ⁴Houmo AI
† Equal contribution, *Corresponding author

---

Overview

MVCTrack is an enhanced framework for **3D single object tracking (3D SOT)** in point clouds, designed to address the limitations of sparse and incomplete LiDAR data. Our approach introduces a **Multimodal-guided Virtual Cues Projection (MVCP)** scheme to enrich sparse point clouds by integrating RGB camera data, significantly improving tracking performance, particularly in scenarios with distant or small objects.

This repository provides the code for MVCTrack, which achieves state-of-the-art performance on the NuScenes dataset.

---

Video

• Please watch this video to know more about our work.

Key Features

Multimodal-guided Virtual Cues Projection (MVCP)

A novel, lightweight, and plug-and-play scheme that:

1. Utilizes 2D object detection to generate virtual cues from RGB images.
2. Projects dense 2D semantic information into 3D space to balance point cloud density.
3. Enhances the sparsity and completeness of point clouds.

MVCTrack Framework

An end-to-end 3D SOT tracker that:

1. Seamlessly integrates the MVCP scheme to improve tracking accuracy.
2. Effectively balances point density distribution across different distances.
3. Achieves competitive performance with minimal computational overhead.

State-of-the-Art Performance

• Evaluated on the large-scale NuScenes dataset.
• Significantly surpasses existing multi-modal 3D trackers.
• Demonstrates exceptional performance in sparse and occluded scenarios.

---

Quick Start

Here are the quick links to the detailed guides:

• Dataset Preparation: Instructions to prepare datasets like KITTI, NuScenes, and Waymo Open Dataset.
• Installation Guide: Step-by-step guide to set up the environment and install dependencies.
• Testing the Model: How to evaluate the pretrained model on your dataset.
• Training the Model: Guide to train the model on your dataset.
• Virtual Cues: Details about the Multimodal-Guided Virtual Cues Projection (MVCP) module.

---

Acknowledgement

Our implementation is based on Open3DSOT, BEVTrack, P2P, MMDetection3D, and MVP. Thanks for the great open-source work!

Citation

If any parts of our paper and code help your research, please consider citing us and giving a star to our repository.