Paper Slam 4/22: When 3D Scenes Meet Video Streams — InHabit vs. CoInteract
> Feynman-style framing: two papers both address "putting humans into environments," but one asks "how do we train AI after placing them," the other asks "how do we show this to humans." Different starting points, completely different solutions.
Overview
Both papers, submitted to arXiv within 24 hours of each other, answer the question "how do we place humans into scenes" — yet they fight on entirely different battlefields.
- InHabit (arXiv:2604.19673): generates large-scale, physically plausible, semantically rich "human-in-3D-scene" training data for embodied agents.
- CoInteract (arXiv:2604.19636): generates consumer-facing video of a person holding a product, with stable hands and physically consistent contact — for e-commerce and virtual marketing.
- Problem: real mocap datasets (PROX, RICH, PiGraphs) are accurate but tiny; synthetic ones (BEDLAM, HUMANISE) scale but are semantically poor (a person boxing in a living room).
- Pipeline — Render-Generate-Lift: 1. *Render*: scene-aware camera sampling renders RGB/depth from 3D scenes (e.g., HM3D). 2. *Generate*: a VLM (Gemini) performs affordance reasoning ("someone sits on the sofa watching TV"); an image-editing model generates scene-adaptive people into the image. 3. *Lift*: monocular depth (MoGe) initializes placement; SMPL-X parameters (pose θ, shape β, translation t, explicit scale s) are optimized with projection, penetration, contact (PhySIC-inspired), and prior losses against the full scene SDF.
- Quality control: four filters — depth-boundary (detects scene geometry tampering), body size (1.2–2.2 m), penetration, and contact plausibility (DECO) — retain ~80% of samples.
- Output: the InHabitants dataset — 78,000+ samples, ~800 building-scale scenes, single- and multi-person interactions.
- Results:
- Contact estimation: augmenting DECO's training improves DAMON benchmark performance without any manual labels.
- HSI reconstruction: Human3R gains mostly on scene contact metrics (V2S: 200.35→176.44; F1: 0.268→0.499); GRAFT improves on both pose and contact, and generalizes to outdoor RICH scenes despite zero outdoor training data.
- Human preference: preferred over GenZI/POSA in 78% of comparisons.
- Essence: knowledge distillation from 2D foundation models into a 3D dataset. Limitation: static poses only, not dynamics (authors note video extension as future work).
- Problem: RGB supervision is too indirect; hands deform, fingers merge, hands penetrate objects in DiT-based video models (Wan, HunyuanVideo, SkyReels).
- Human-Aware MoE: detector boxes (MediaPipe, Face Mesh) provide spatial supervision for a router that routes hand/face tokens to lightweight dedicated experts; general tokens go to a shared expert.
- Spatially-Structured Co-Generation:
- An auxiliary HOI stream encodes interaction geometry: body mesh projections + SAM3 object masks fused into a texture-free 3-channel structure map.
- Both streams share a DiT backbone with separate patch embeddings and adaptive layer-norm parameters.
- 3D RoPE: streams are concatenated along the width axis (RGB at w ∈ [0, W], HOI at w ∈ [-W, 0]); time axis encodes history frames (t < 0), generation window (t ∈ [0, T]), and reference image anchors.
- Asymmetric co-attention: Stage 1 is bidirectional; Stage 2 masks HOI keys from RGB queries, so the HOI branch can be dropped entirely at inference with zero overhead — the geometry prior stays baked into shared weights.
- Data pipeline: Qwen-Edit decouples person/product references, validation filters mismatched triplets, SAM3 segments objects, SAM3D-body recovers meshes.
- Results: outperforms AnchorCrafter and InteractAvatar on unseen objects in structural stability, logical consistency, and interaction realism.
- Essence: implicit structural supervision — geometry constraints are learned during training rather than enforced by per-sample optimization.
- Bias inheritance: InHabit's affordances inherit VLM training biases (under-represented behaviors never get proposed).
- Cargo-cult physics?: CoInteract learns statistical regularities of "no penetration" — does it *understand* contact, or just avoid unseen pixel patterns?
- Possible convergence: use InHabit-style placement to produce 3D pose conditions, then CoInteract-style co-generation (perhaps a triple-stream RGB + HOI + 3D pose design) to animate them — a potential data flywheel between the two.
- Neither system yet achieves simultaneous physical, semantic, visual, and temporal plausibility; both are engineering achievements, not evidence of genuine "understanding" of human interaction.
- Kister et al., "InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement", arXiv:2604.19673, 2026
- Luo et al., "CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation", arXiv:2604.19636, 2026
- Pavlakos et al., "Expressive Body Capture" (SMPL-X), CVPR 2019
- Peebles & Xie, "Scalable Diffusion Models with Transformers" (DiT), ICCV 2023
- Hassan et al., "Resolving 3D Human Pose Ambiguities with 3D Scene Constraints" (PROX), ICCV 2019
- Tripathi et al., "DECO: Contact Estimation from 3D Human Scenes", CVPR 2023
Key points
InHabit — distilling internet common sense into 3D data
CoInteract — fixing structural collapse in interaction video
Head-to-head
| Dimension | InHabit | CoInteract | |---|---|---| | Consumer | models (research pipelines) | humans (shoppers, viewers) | | Output | static 3D poses | dynamic video | | Physics | explicit optimization (L_pen, L_contact) | implicit, learned from HOI stream | | Supervision | 2D foundation models (VLM common sense) | 3D structure streams from real geometry | | Architecture | staged pipeline (swappable, error-prone) | end-to-end (globally optimal, black box) |