MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction

Xiaohao Xu*, Feng Xue*, Shibo Zhao, Yike Pan, Sebastian Scherer, Xiaonan Huang

University of Michigan, Ann Arbor & Carnegie Mellon University

The full code is available! 😃

❤️ Video Demo ⭐ ArXiv Paper

Pipeline: Multi-Agent Gaussian Consensus 🙌

MAC-Ego3D leverages parallel Intra-Agent Gaussian Consensus and periodic Inter-Agent Gaussian Consensus to enable real-time pose tracking and photorealistic 3D reconstruction using a shared 3D Gaussian map representation.

---

Environment Setup

To set up the required environment, follow the following steps. Alternatively, you can set up the environment using the Dockerfile.

0. Git clone the repo with submodules:

   git clone --recursive https://github.com/Xiaohao-Xu/MAC-Ego3D.git

1. Create and activate a new Conda environment:

   conda create -n macego python==3.9
   conda activate macego

2. Install necessary dependencies:

   conda install -c "nvidia/label/cuda-11.8.0" cuda-toolkit
   conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
   pip install -r requirements.txt

3. Install PCL for fast-gicp submodule:

   pip install pcl
   sudo apt install libpcl-dev
   sudo apt install libusb-1.0-0-dev

4. Install additional submodules:

   conda activate macego
   pip install submodules/diff-gaussian-rasterization
   pip install submodules/simple-knn

5. Build and install the fast_gicp submodule:

   cd submodules/fast_gicp
   mkdir build
   cd build
   cmake ..
   make
   cd ..
   python setup.py install --user

6. Download model weights for the salad submodule:

   Follow the link to download the weights:
   Loop Closure Detection Model Weights

   After downloading, place the weights in the submodules/salad directory.

---

Datasets

Multi-Agent Replica

1. Download the Multi-Agent Replica dataset:

   Please download the dataset from the HuggingFace Repo and unzip them.

2. Adjust directory structure:

   Update the directory structure to match the following:

   Original Structure:

   Replica
      - {scene_name}
        - {scene_name_agent_id}
          - results (contain rgbd images)
              - frame000000.jpg
              - depth000000.png
              ...
          - traj.txt

   Required Structure:

   Replica
      - {scene_name}
        - {scene_name_agent_id}
           - images (contain rgb images)
               - frame000000.jpg
               ...
           - depth_images (contain depth images)
               - depth000000.png
               ...
           - traj.txt

7Scenes Dataset

1. Download the 7Scenes dataset:
   7Scenes Dataset

2. Configure the dataset:
   Some sequences in certain scenes may have missing frames. Make sure to remove any empty folders after processing.

   python ./utils/7Scene2ICP.py  # Modify the script to reflect the correct dataset and output paths

---

Running the Code

Multi-Agent Replica Dataset

To run all the experiments on the Multi-Agent Replica dataset:

bash multitest_multi_agent_replica.sh

7Scenes Dataset

To run all the experiments on the 7Scenes dataset:

bash multitest_7scene.sh

---

Citation

If you find this repository useful, please cite our paper:

@article{xu2024macego3d,
  title={MAC-Ego3D: Multi-Agent Gaussian Consensus for Real-Time Collaborative Ego-Motion and Photorealistic 3D Reconstruction},
  author={Xu, Xiaohao and Xue, Feng and Zhao, Shibo and Pan, Yike and Scherer, Sebastian and Huang, Xiaonan},
  journal={arXiv preprint arXiv:2412.09723},
  year={2024}
}

Contact

For questions or further inquiries, please report an issue or reach out to: Xiaohao Xu
Email: xiaohaox[at]umich.edu