Open-CHOIR: Open-world Contact-aware 4D Hand-Object Interaction Reconstruction

We ask whether everyday open-world monocular videos can be turned into reusable 4D interaction primitives: articulated hand motion, object shape with 6D pose over time, and the when/where of contact. Such a capability would enable scalable mining of real interactions and, beyond reconstruction, support scene-aware synthesis and planning. However, reconstructing hand-object interaction (HOI) from challenging monocular videos remains difficult: methods often assume known objects or curated scenes, and separately estimated hands and objects easily become misaligned under clutter, occlusion, and unseen object geometries. Targeting this setting, we present Open-CHOIR, an Open-world Contact-aware HOI Reconstruction framework for a monocular camera, using contact as an explicit coupling signal between hands and objects. Open-CHOIR first initializes a coarse, contact-agnostic 4D HOI sequence from open-world visual priors. It then introduces a generative HOI spatial rectification module to predict ray-depth corrections and rectify hand-object relative placement, then derives initial per-frame contact correspondences on the rectified geometry. Last, a contact-aware joint optimization with dynamically updated contact constraints enforces geometric, temporal, and contact consistency. Experiments on controlled and challenging videos show that Open-CHOIR improves object reconstruction, physical plausibility, and temporal consistency over state-of-the-art methods.

Contributions

An open-world monocular 4D HOI reconstruction method that extracts 2D interaction cues, performs guarded object tracking, and produces a coarse contact-agnostic sequence.
A generative HOI spatial rectification module that predicts ray-depth corrections before constructing initial per-frame barycentric contact correspondences.
A contact-aware joint optimization with dynamically updated contact constraints, validated on controlled and in-the-wild videos.

Method Overview

Stage 1 coarse 4D HOI initialization — From a monocular video, we first obtain 2D HOI cues and initialize 3D hand and object reconstructions to form isolated sequences.

Stage 2 generative HOI spatial rectification — A flow-matching generative HOI spatial rectification module rectifies hand-object relative placement before initial contact correspondence construction.

Stage 3 hierarchical contact-aware optimization — Contact-aware optimization refines the sequence across phases, yielding physically consistent 4D HOI sequences.

Our Representative Examples

1 / 0

Explore representative sequences with synchronized input video, reconstructed 4D geometry, and rest-pose contact predictions. Click on the above arrows to browse different representative examples.

Qualitative Comparison with SOTA Methods

More Qualitative Results of Our Method

Filter by click to switch

Bimanual Cases

Our method generalizes to challenging bimanual scenarios — two-hand two-object and two-hand one-object hand–object interaction. Drag the 3D viewer to explore the reconstruction from any angle.

Acknowledgements

This work was partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N, the CUHK Direct Grant for Research under Project 4055296, and gifts from Adobe and the UCL AI Centre.

BibTeX

@article{openchoir2026,
  title     = {Open-CHOIR: Open-world Contact-aware 4D Hand-Object Interaction Reconstruction},
  author    = {Xu, Hao and Liu, Yilin and Wang, Yinqiao and Fu, Chi-Wing and Mitra, Niloy J.},
  journal   = {ACM Transactions on Graphics},
  volume    = {45},
  number    = {6},
  year      = {2026},
  publisher = {Association for Computing Machinery},
  note      = {Presented at SIGGRAPH Asia 2026}
}