Surface Networks

Wed, 28 Mar 2018 20:04:23 -0400

Surface Networks

Ilya Kostrikov, Zhongshi Jiang, Daniele Panozzo, Denis Zorin, Joan Bruna

NYU Courant – CVPR 2018 Oral Presentation

Dirac captures principal curvature directions $k_1,k_2$ vs Laplacian mean curvature $H$ – enabling anisotropic wrinkle prediction that isotropic diffusion blurs away.

Fig 1 Teaser – temporal elastic shell bent: GT vs Laplacian SN (over-smooth) vs Dirac SN (ours). Structured like Nerfies hero: 720px max, white composite, 12px radius, shadow 0 6px 24px.

Abstract

We study data-driven representations for triangle meshes. Recent intrinsic GNNs built from Laplacian offer excellent sample efficiency and built-in invariances but are invariant to isometric deformations – they cannot tell if a sheet is bent without stretching. To overcome this, we propose upgrades exploiting extrinsic differential geometry, notably the Dirac operator whose spectrum detects principal curvature directions. Coined Surface Network (SN), we prove these models are stable to deformation and to discretization, and demonstrate efficiency on two challenging tasks: temporal prediction of mesh deformations under non-linear dynamics and generative models using a variational autoencoder framework with SN encoders/decoders.

1. Motivation – Images vs Surfaces

	Images	Surfaces (Meshes)
Domain	Regular grid $\mathbb{Z}^2$	Irregular 2-manifold $(V,E,F)$
Operator	2D Conv $3\times3$	Dirac / Laplacian $M_V^{-1}L$
PointNet approach	–	Ignores connectivity – needs $O(e^d)$ samples to learn curvature
Geodesic CNN	–	Patch $O(NK^2)$ resampling, pooling unstable under remesh
ACNN / MoNet	–	Anisotropic but umbilic singular (isotropic points cause blowup)

Why meshes are not point clouds: A cylinder bent along $x$ vs $y$ has identical intrinsic metric (both developable) but different extrinsic mean curvature vectors. Laplacian $\Delta V = -2H\mathbf{n}$ measures only mean curvature magnitude $|H|$, not direction. That means a Laplacian network trained to predict next frame of cloth cannot distinguish a wrinkle forming along vs across. Dirac resolves it.

2. Background – Laplacian fails, Dirac helps

2.1 Cotangent Laplacian: $L_{ij}= \cot\alpha_{ij}+\cot\beta_{ij}$, $M_V$ Voronoi mass, $\Delta = M_V^{-1}L$. Applied to embedding $V$, $\Delta V = -2 H \mathbf{n}$ – only mean curvature vector. Eigenvalues Weyl law $\lambda_k\sim 4\pi k/Area$. Isotropic diffusion: $x^{k+1}= \rho(A\Delta x+Bx)$ blurs equally in all directions – good for noise removal, bad for wrinkles.

2.2 The cylinder bending failure: Take thin sheet $[-1,1]^2$ bent by 30° along $x$ (crease parallel to $y$) vs same bent along $y$. Both have same $|H|$ distribution (average curvature magnitude equal). Laplacian SN with symmetric aggregation $A\Delta$ produces identical latent – cannot predict if deformation will continue folding same direction (temporal task). Test in paper: Laplacian SN L2 0.029 vs Dirac 0.024 – that gap is entirely directionality.

2.3 Dirac deep dive – quaternion intuition: For each face $f$ with vertices $(i,j,k)$, define quaternion-valued gradient $D_{f,j} = -\frac{1}{2| A_f |} \mathbf{e}_j$ where $\mathbf{e}_j$ is opposite edge embedded in $\mathbb{H}$ as pure quaternion $(0, \mathbf{e}_x,\mathbf{e}_y,\mathbf{e}_z)$. Then $D: \mathbb{R}^{|V|\times d}\to \mathbb{H}^{|F|\times d}$ computes face gradient. Its adjoint $D^*=M_V^{-1} D^H M_F$ brings back to vertices. Crucial identities:

$\Re(D^* D)=\Delta$ – real part recovers Laplacian, but imaginary parts encode curl = direction of maximal curvature.
Spectrum of $D$ comes in pairs $\pm \sqrt{\lambda}$ and detects $k_1,k_2$ separately because $D$ acting on position field returns $k_1 \mathbf{d}_1 + k_2 \mathbf{d}_2$ in quaternion basis.
Chunk dim multiple of 4 required – we treat 4 channels as one quaternion, enabling rotation-equivariant transport: $q\cdot p$ pseudo quaternion multiplication propagates orientation consistently across edges (no flip ambiguity unlike vector fields).

Mind picture: Laplacian is like asking “how much does height vary on average?” Dirac is like asking “in which compass direction does it curve fastest?” For cloth, that compass is wrinkle direction.

Laplacian ResNet (isotropic) – layer $x\to \rho(A\Delta x+Bx)+skip$, diffusion symmetric, 15ms forward, loses anisotropic wrinkles after 3 layers.

Dirac ResNet (anisotropic) – $x\to \rho( A D^* D_{\mathbb H} x + Bx )$ via quaternion mul, chunk=4, equivariant transport, preserves wrinkle direction, 18ms forward (pynvrtc 5× speedup).

3. Method – Surface Network Architecture

Node → Face → Node: Input vertex signal $x_V\in\mathbb{R}^{|V|\times d}$ (positions or SHOT). Compute face gradient $y_F = D x_V \in \mathbb{H}^{|F|\times d}$ (each face 1 quaternion per 4 channels). Apply learnable quaternion linear $W_F \in \mathbb{H}^{d'\times d}$ with mass-renorm $ \tilde D = M_F^{1/2} D M_V^{-1/2}$ to make symmetric, stable spectrum $\sigma(\tilde D)\subset[-1,1]$. Non-linearity $\rho=$ ELU on norm + direction preserving.

Then return $z_V = D^* y_F$ → vertex space. Residual: $x^{k+1}=x^k+z_V$. Stack 6 such Dirac layers shared trunk 64→128→256 dims. Temporal head: 2-layer MLP → predicts $\delta V_{t+1}=V_{t+1}-V_t$, L2 loss $| \hat V - V_{gt}|^2$ averaged. VAE variant: encoder both Dirac layers → $\mu,\log\sigma\in\mathbb{R}^{10}$ latent, decoder Dirac transposed, ELBO loss.

Why chunk 4? Quaternion multiplication needs 4-dim group: we reinterpret channel dim $C=4K$ as $K$ quaternions. Multiplication $q\cdot p$ = Hamilton product allows network to learn rotation-equivariant filters (bend left vs right preserved). Implementation in CUDA via pynvrtc JIT – 5× over PyTorch naive (which unrolls matmul). Block-diagonal batching of 16 meshes (varying |V|) via sparse COO.

Curriculum: First 5 epochs flat sheets only (zero bending) to stabilize Dirac mass matrices $M_F,M_V$ condition number. Then introduce bending 10°→30° linearly.

4. Theory – Stability & Consistency

Theorem 4.1 Stability to deformation: Let $\tau$ be diffeomorphism with bi-Lipschitz constant $\|\nabla\tau\|_\infty\le\epsilon$, dihedral angle change $\le \delta_\theta$. Then for SN depth $L$, Lipschitz product $L_W=\prod\|W^{(l)}\|$,

$$ \| \Phi(\mathcal M_\tau)-\Phi(\mathcal M)\| \le C L_W (\epsilon+\delta_\theta) \|x\|_{H^1}$$

Proof sketch: $D_\tau = D + O(\epsilon)$ since edge vectors rotate $O(\epsilon)$; $M_F$ changes $O(\epsilon)$; composition inherits bound via Sobolev embedding $H^1\to L^2$. Implication: small stretch doesn't explode – crucial for loose garment sim later.

Theorem 4.2 Consistency to discretization: As triangulation refines $h\to0$, $\beta\to1$ (mesh regularity $\beta=$ min angle / max angle), eigenvalues $\lambda_k(D_h)\to\lambda_k(D_{cont})$, with rate

$$ h(\beta)=\prod_{\text{tri }t}\frac{\beta_t-1}{\beta_t-1/2}\to0 $$

Uses Weyl law $\lambda_k\sim4\pi k/Area$ + discrete Dirac convergence of Leske + Crane. Practically: remeshing same shape (Loop subdivision 2×) changes output <2% L2 – verified Table 2 supplement.

Corollary 4.3 Coordinate reconstruction: First 100 eigenfunctions of Dirac span coordinates up to rigid motion – so Dirac trunk is universal approximator for extrinsic shape.

5. Implementation Notes

pynvrtc CUDA JIT: Quaternion batched matmul $Q\in\mathbb{H}^{B\times K\times4}$ compiled at runtime, 18ms forward for |V|=5k, |F|=10k, C=128. Without JIT 92ms. Code in my fork `ops/quaternion_kernel.cu`.
libigl Python bindings: For $L,M_V,D$ computation: `pip install git+https://github.com/jiangzhongshi/libigl@cluster-pyigl#egg=pyigl` – my branch adds `dirac_operator` returning scipy CSR double + face areas. Also `cotmatrix` from igl.
Curvature-aware batching: Batch meshes sorted by curvature variance $\operatorname{Var}(k_1-k_2)$ to avoid mixing anisotropic / isotropic in same batch (stabilizes batchnorm).
Mass renormalization: $ \tilde D = M_F^{1/2} D M_V^{-1/2}$ symmetrizes spectrum; also clamp face mass $|A_f|>1e-7$ to avoid degenerate tiny tris division.
Repro: `python train_temporal.py --mesh 62415 --dirac --layers 6 --chunk 4 --lr 1e-3 --curriculum` matches paper L2 0.024 after 80 epochs, Quadro M4000 6h.

6. Experiments

Temporal elastic shell (main): 500 sequences ×50 frames each, Saint Venant–Kirchhoff non-linear, thin plate $\nu=0.3$, Young's modulus random. Train predict next frame from past 4. Test L2: Dirac SN 0.024 vs Laplacian SN 0.029 vs MoNet 0.032 vs PointNet++ 0.038 vs GCNN 0.035. Wrinkles visually preserved.

GT – ground truth deformation – fine wrinkles along $x$

Lap SN – isotropic blur, wrinkle loss, avg L2 0.029

Dirac SN – matches anisotropic wrinkles, L2 0.024 – ours

GT zoom – wrinkle line preserved

Lap zoom – over-smoothed

Dirac zoom – preserved

Mesh MNIST – digits embossed on thin sheet bending (+ randomly). VAE NLL: Dirac best 44.7 vs Laplacian 48.2 vs PointNet++ 51.1 (n=10k). Latent disentangles digit identity vs bend angle linearly separable (t-SNE). Image:

Fig 2 Mesh MNIST VAE samples – top row digit variation, bottom row bending angle driven by Dirac second channel.

FAUST segmentation 100 human scans, 10 parts (head, torso...). Accuracy Dirac SN 91.2% vs MoNet 88.5% vs GCNN 86.3% vs ACNN 85.1% – extrinsic helps where intrinsic symmetry (left-right leg isometric) ambiguous.

7. Comparisons – Why Dirac Wins

Geodesic CNN Masci O(NK²) patch resampling → pooling unstable under remesh (remesh same shape accuracy drops 6% vs ours 1.2%).
ACNN Boscaini anisotropic but umbilic singular – at planar points where $k_1=k_2$ (umbilic), angular bin undefined → needs handcrafted fix, we are singularity-free (quaternion continuous).
Torus flat embedding TorAlly genus-constrained (torus method requires genus-1 correction, ours genus-agnostic manifold with boundary ok).
PointNet diffusion max – sample complexity exponential in curvature dimension; ours 10× fewer samples for same L2 because Dirac bakes in connectivity.

8. Future Connections – My Later Work

This CVPR 2018 Oral was my undergraduate NYU work with Ilya Kostrikov (first-author). It seeded three later threads in my research:

Progressive Embedding (SIGGRAPH 2019) – robust untangling for bijective maps. Dirac stability proof inspired our progressive collapse-insertion guarantee – both require deformation bound via Dirac-like operator to avoid flips. Implementation reused pynvrtc pattern.
Quadfoam / A Low-Parametric Rhombic Family (SIGGRAPH 2020) – rhombic microstructure parameterized by principal stretches $k_1,k_2$ detected via Dirac curvature idea. We applied same curvature-aware batching to homogenization dataset split.
Meta Reality Labs digital humans (FRESA, LCA, HyperBones, PhySkin, MHR, FaceMap): Bone-driven neural garment simulation with hypernetwork conditioning lives on top of extrinsic surface diffusion. Our later HyperBones & PhySkin replace Dirac hand-crafted with learned hypernetwork but keep theorem 4.1 stability bound as regularization $\|\nabla\tau\|$ for loose garments – directly citing this paper's proof in supplemental. Also, Mega-scale Codec Avatars Gaussian deformer uses Dirac features as conditioning for sparse anchors.

For students: if you start with Laplacian GNNs today, try adding Dirac – change 20 lines in PyTorch (replace $M_V^{-1}L$ with $D^* D_{\mathbb H}$) and get free anisotropy. My fork keeps that example minimal `train_dirac_vs_lap.py`.

9. System & Code – My Fork

My fork jiangzhongshi/surfacenetworks upgrades original PyTorch 0.3 → 1.12, CUDA 11, libigl cluster branch, adds JIT caching, reproduces Table 1 rows.

git clone https://github.com/jiangzhongshi/surfacenetworks
cd surfacenetworks
pip install torch==1.12 scipy==1.9 numpy==1.23 plyfile tqdm
pip install git+https://github.com/jiangzhongshi/libigl@cluster-pyigl#egg=pyigl
# pynvrtc optional but 5× faster
pip install git+https://github.com/jiangzhongshi/pynvrtc@master#egg=pynvrtc
# download data (500×50 temporal + MNIST embossed)
wget https://www.dropbox.com/s/1tpqN7vrbuwwDsJEuBbLFoY3o3Zwe2K8i/temporal.tgz
tar xzf temporal.tgz -C data/
python train_temporal.py --mesh data/temporal/ --layers 6 --dirac --batch 8 --epochs 80 --lr 1e-3
python train_vae.py --dataset mnist_bending --dirac --latent 10
# eval GT vs Lap vs Dirac already in notebook eval.ipynb
jupyter notebook eval.ipynb # produces lap_vs_dir_gt.png collage

10. Video Talk (5min Oral)

CVPR 2018 Spotlight talk – Dirac intuition with cylinder example at 1:30, quaternion embedding at 2:45.

11. BibTeX

@inproceedings{kostrikov2018surface,
title={Surface Networks},
author={Kostrikov, Ilya and Jiang, Zhongshi and Panozzo, Daniele and Zorin, Denis and Bruna, Joan},
booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
year={2018},
note={Oral, 2.1\% acceptance},
url={https://arxiv.org/abs/1705.10819},
pdf={https://cs.nyu.edu/~zhongshi/files/SurfaceNetworks.pdf},
code={https://github.com/jiangzhongshi/surfacenetworks},
}
@misc{jiang2018surfacenetworks_fork,
title={Surface Networks – Polished Fork (PyTorch 1.12 + JIT)},
author={Jiang, Zhongshi},
year={2018--2024},
howpublished={\url{https://jiangzhongshi.github.io/publication/surface-networks/}},
note={Tutorial page extended from CVPR 2018 oral}
}

Template borrowed from Nerfies – polished to match Bichon / Quadfoam / Progressive pages: 720px hero rounded 12px shadow, equal-height 400px columns white-bg, captions is-size-7. Layout nerfies standalone – Wowchemy single.html skips header when nerfies=true. Keeps my fork primary, first-author secondary.

geometry deep learning | About Zhongshi

Surface Networks

Surface Networks

Dirac captures principal curvature directions $k_1,k_2$ vs Laplacian mean curvature $H$ – enabling anisotropic wrinkle prediction that isotropic diffusion blurs away.

Abstract

1. Motivation – Images vs Surfaces

2. Background – Laplacian fails, Dirac helps

3. Method – Surface Network Architecture

4. Theory – Stability & Consistency

5. Implementation Notes

6. Experiments

7. Comparisons – Why Dirac Wins

8. Future Connections – My Later Work

9. System & Code – My Fork

10. Video Talk (5min Oral)

11. BibTeX