📜

Neuralangelo

Key phrases from the abstract

•
multi-resolution 3D hash grids
•
neural surface rendering
•
numerical gradients for computing higher-order derivatives as a smoothing operation
•
coarse-to-fine optimization on the hash grids

1. Introduction

•
Neural surface reconstruction methods
What
Pros: continuity
Cons: scalability
•
Neuralangelo
◦
Based on
▪
Instant NGP as a neural SDF representation
◦
+ new findings for better optimization
1.
Stabilizing the optimization
a.
numerical gradients for computing higher-order derivatives
i.
e.g. surface normals for the eikonal regularization
2.
Progressive optimization (=coarse-to-fine)
a.
Better details
◦
Scope
▪
reconstruction accuracy
▪
view synthesis quality

2. Related work

•
Multi-view surface reconstruction
•
Neural Radiance Fields (NeRF)
◦
How an isosurface of the volume density could be defined to represent the underlying 3D geometry
•
Neural surface reconstruction
◦
auxiliary informations
▪
patch warping
▪
sparse point clouds
▪
depth
▪
segmentation
▪
scene representations with hash encodings

3. Approach

3.1. Preliminaries

Graphics에서 volume rendering에 대한 기초: https://youtu.be/y4KdxaMC69w 핵심 질문들: 뷰포인트가 달라도 어떻게 샘플링 간격을 일정하게 유지하는지 왜 뒤 쪽에서부터 렌더하는지
CUDA 빼고 순수 Python으로 구성한 Instant NGP: https://velog.io/@janet990430/Instant-NGP-NeRF-개발-도전기

Neural volume rendering (NeRF)

c^(o,d)=∑i=1NTiαici\hat c(\mathbf o, \mathbf d) = \sum ^N _{i=1} T_iα_i \mathbf c_i
•
the accumulated transmittance TiT_i
◦
the fraction of light that reaches the camera
Ti=∏j=1i−1(1−αj)T_i = \prod^{i−1}_{j=1}(1 − α_j)
•
color ci\mathbf c_i
•
opacity αiα_i
αi=1−exp⁡(−σiδi)α_i = 1 − \exp(−σ_iδ_i)
•
volume density σiσ_i
•
the distance between adjacent samples δiδ_i
δi=ti+1−tiδ_i = t_{i+1} − t_i
•
color loss: rendered image c^\hat {\mathbf c} vs. input image c\mathbf c
LRGB=∣∣c^−c∣∣1\mathcal L_\text{RGB} = ||\hat {\mathbf c} - {\mathbf c}||_1

Volume rendering of SDF

αi=max⁡(Φs(f(xi))−Φs(f(xi+1))Φs(f(xi)),0)α_i = \max ({\Phi_s(f(\mathbf x _i)) - \Phi_s(f(\mathbf x _{i+1})) \over \Phi_s(f(\mathbf x _i)) }, 0)

Multi-resolution hash encoding

γ(xi)=(γ1(xi,1),...γL(xi,L))γ(\mathbf x_i) = (γ_1(\mathbf x_{i, 1}), ... γ_L(\mathbf x_{i, L}))

3.2. Numerical Gradient Computation

Leik=1N∑i=1N(∥∇f(xi)∥2−1)2\mathcal L_\text{eik} = {1\over N} \sum^ N _{i=1}(∥∇f(\mathbf x_i)∥_2 − 1)^2
γl(xi,l)=γl(⌊xi,l⌋)⋅(1−β)+γl(⌈xi,l⌉)⋅βγ_l(\mathbf x_i,l) = γ_l(⌊\mathbf x _{i,l}⌋) · (1 − β) + γ_l(⌈\mathbf x _{i,l}⌉) · β
∂γl(xi,l)∂xi=γl(⌊xi,l⌋)⋅(−∂β∂xi)+γl(⌈xi,l⌉)⋅∂β∂xi=γl(⌊xi,l⌋)⋅(−Vl)+γl(⌈xi,l⌉)⋅Vl{∂_{γl}(\mathbf x_{i,l}) \over ∂ \mathbf x_i} = γ_l(⌊\mathbf x_{i,l}⌋) · (− {∂β \over ∂ {\mathbf x_i}} ) + γ_l(⌈\mathbf x_{i,l}⌉) · {∂β \over ∂{\mathbf x_i}} \\= γ_l(⌊\mathbf x_{i,l}⌋) · (−V_l) + γ_l(⌈\mathbf x_{i,l}⌉) · V_l
∇xf(xi)=f(γ(xi+ϵx))−f(γ(xi−ϵx))2ϵ∇_x f(\mathbf x_i) = {f (γ(\mathbf x_i + \mathbf ϵ_x)) − f (γ(\mathbf x_i − \mathbf ϵ_x)) \over 2ϵ }

3.3. Progressive Levels of Details

3.4. Optimization

Lcurv=1N∑i=1N∣∇2f(xi)∣\mathcal L_\text{curv} = {1 \over N} \sum^N_{i=1} |∇^2f(\mathbf x_i)|
L=LRGB+weikLeik+wcurvLcurv\mathcal L = \mathcal L_\text{RGB} + w_\text{eik} \mathcal L_\text{eik} + w_\text{curv} \mathcal L_\text{curv}