Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera

Mirgahney Mohamed · Lourdes Agapito

3DV 2024

Paper | Project Page | Poster

We propose DynamicSurf, a model-free neural implicit surface reconstruction method for high-fidelity 3D modelling of non-rigid surfaces from monocular RGB-D video.

Usage

Data Convention

The data is organized as NeuS

<case_name>
|-- cameras_sphere.npz    # camera parameters
|-- depth
    |-- # target depth for each view
    ...
|-- image
    |-- # target RGB each view
    ...
|-- mask
    |-- # target mask each view (For unmasked setting, set all pixels as 255)
    ...

Here cameras_sphere.npz follows the data format in IDR, where world_mat_xx denotes the world-to-image projection matrix, and scale_mat_xx denotes the normalization matrix.

Pre-processed Data

You can download a part of pre-processed KillingFusion data here and unzip it into ./.

Important Tips: If the pre-processed data is useful, please cite the related paper(s) and strictly abide by related open-source license(s).

Setup

Clone this repository and create the environment (please notice CUDA version)

git clone https://github.com/Mirgahney/dysurf.git
cd dysurf

The code is tested with Python 3.9 and PyTorch 1.11 with CUDA 11.3. DynamicSurf requires smooth_sampler, which is a drop-in replacement for PyTorch's grid sampler that support double back-propagation. Also the following packages are required:

conda env create -f dysurf.yml
conda activate dysurf

Dependencies (click to expand)

• torch==1.11
• pytorch3d
• scikit-image
• open3d
• imageio
• matplotlib
• configargparse
• tensorboard
• opencv-contrib-python
• opencv_python==4.5.2.52
• trimesh==3.9.8
• numpy==1.21.2
• scipy==1.7.0
• PyMCubes==0.1.2

Running

• Training

python3 -W ignore exp_runner_grid_hydra.py 

• Evaluating pre-trained model

  Coming Soon

Data Pre-processing

Coming Soon

Geometric Projection

Coming Soon

Todo List

• Code of Data Pre-processing
• Code of Geometric Projection
• Pre-trained Models and Evaluation Code
• Training Code

Acknowledgements

This project is built upon NDR, Go-Surf, and NeuS. Some code snippets are also borrowed from IDR and NeRF-pytorch. The pre-processing code for camera pose initialization is borrowed from Fast-Robust-ICP. The evaluation code for geometry rendering is borrowed from StereoPIFu_Code. Thanks for these great projects. We thank all the authors for their great work and repos.

Contact

If you have questions, please contact Mirgahney Mohamed.

Citation

If you find our code or paper useful, please cite

@misc{mohamed2023dynamicsurf,
      title={DynamicSurf: Dynamic Neural RGB-D Surface Reconstruction with an Optimizable Feature Grid},
      author={Mirgahney Mohamed and Lourdes Agapito},
      year={2023},
      eprint={2311.08159},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

If you find our pre-processed data useful, please cite the related paper(s) and strictly abide by related open-source license(s).