Paper Slam 4/22: When 3D Scenes Meet Video Streams — InHabit and CoInteract Compared
> Feynman's perspective: both papers tackle "placing a person into an environment," but one asks "how to train AI afterwards" and the other asks "how to show it to humans." Different starting points, completely different solutions.
Introduction: Two Questions About "Humans in Scenes"
Walk into an empty apartment. There's a chair, a table, a lamp. You would sit down — because chairs are for sitting. No one taught you; you internalized "how humans use space" from countless internet images.
Two research teams, in the same week (April 21, 2026), submitted two arXiv papers answering "how to place a person into a scene," from completely different angles:
- InHabit (2604.19673): How to mass-generate "humans in 3D scenes" data for training embodied agents. Goals: scale, physical plausibility, semantic richness. Pipeline: render-generate-lift.
- CoInteract (2604.19636): How to generate video of a person holding a product while talking, with realistic hand-object contact. Goals: structural stability (no deformed hands), physical consistency (no penetration), visual quality. Tech: Diffusion Transformer (DiT) + dual-stream co-generation + Human-Aware MoE.
- An explicit scale parameter s decouples "how big is this person" from "what is their body shape," stabilizing convergence.
- Using the complete scene mesh/SDF makes penetration/contact checks globally consistent and occlusion handling more robust.
- Contact estimation: augmenting DECO's training with InHabitants (zero manual labels) improves DAMON performance.
- HSI reconstruction: For Human3R, pose (PA-MPJPE 59.25→58.65) slightly improves but scene contact metrics jump (V2S: 200.35→176.44, F1: 0.268→0.499). GRAFT improves further; trained only on InHabitants, it generalizes to RICH's outdoor scenes despite no outdoor training data.
- Perceptual study: InHabitants preferred over GenZI/POSA in 78% of cases.
- RGB stream: standard video diffusion objective.
- HOI stream: auxiliary, geometry-only — body mesh projections fused with object masks (SAM3) into a texture-free 3-channel structure map.
- Stage 1: full bidirectional attention — rapid appearance-structure coupling.
- Stage 2: asymmetric co-attention mask — RGB queries see only RGB keys; HOI queries see both.
- Battlefield: InHabit's consumer is the training pipeline (KPIs: physical precision, alignment, scale); CoInteract's consumer is shoppers/viewers (KPIs: visual quality, stable hands, natural motion).
- Time dimension: static poses vs. dynamic video requiring temporal consistency in VAE latent space.
- Geometry constraints: explicit optimization (precise but slow — seconds to minutes per sample) vs. implicit learning (fast inference, but may fail out-of-distribution).
- Supervision source: 2D foundation model commonsense (generalizes to new scenes via VLM knowledge) vs. 3D structure stream (limited by dataset diversity, though unseen-object results are encouraging).
- Architecture: staged pipeline (controllable, debuggable, error-accumulating) vs. end-to-end training (globally optimized, black-box).
- Users: researchers (open dataset/code) vs. content creators (API/SaaS).
- InHabit: VLM affordances inherit and amplify internet biases (e.g., never proposing floor-sitting if underrepresented). 2D-plausible poses may be 3D-infeasible; the undisclosed failure rate is a blind spot.
- CoInteract: HOI labels come from automatic segmentation/mesh recovery — errors teach wrong geometry. And implicit learning may encode only statistical avoidance of unseen pixel patterns, not physical understanding — *cargo cult physics*.
- Kister et al., "InHabit: Leveraging Image Foundation Models for Scalable 3D Human Placement", arXiv:2604.19673, 2026
- Luo et al., "CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation", arXiv:2604.19636, 2026
- YM et al., "PhySIC: Physics-informed 3D Human-Scene Reconstruction", 2025
- Pavlakos et al., "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image", CVPR 2019 (SMPL-X)
- Peebles & Xie, "Scalable Diffusion Models with Transformers", ICCV 2023 (DiT)
- Rombach et al., "High-Resolution Image Synthesis with Latent Diffusion Models", CVPR 2022 (LDM/VAE)
- Savva et al., "PiGraphs: Learning Interaction Snapshots from Observations", SIGGRAPH 2016
- Hassan et al., "Resolving 3D Human Pose Ambiguities with 3D Scene Constraints", ICCV 2019 (PROX)
- Black et al., "BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion", CVPR 2023
- Tripathi et al., "DECO: Contact Estimation from 3D Human Scenes", CVPR 2023
- Chen et al., "Human3R: Reconstructing 3D Human-Scene from a Single RGB Image", 2025
- Wan et al., "Wan: Open and Advanced Large-Scale Video Generative Models", 2025
One builds training data for models; the other builds consumer content. One obsesses over millimeter-level contact accuracy in 3D; the other over hands not becoming six-fingered blobs in video frames.
Chapter 1: InHabit — Transferring Internet Commonsense into 3D
1.1 The Data Famine
Training embodied agents requires abundant, diverse, 3D data with human behavior — which barely exists. Real datasets (PROX, RICH, PiGraphs) use mocap: high precision, tiny scale (dozens of scenes, limited actions). Synthetic datasets (BEDLAM, HUMANISE) scale up but are semantically poor — a person boxing in a living room, because the motion library contained boxing.
The insight: 2D foundation models (VLMs, image generators) have internalized "how humans use environments" from billions of images. How do we extract that knowledge trapped in 2D image space into trainable 3D data?
1.2 Render-Generate-Lift
Render. Given a 3D scene (e.g., HM3D), sample scene-aware camera views (furniture, activity areas, open sightlines) and render RGB/depth maps.
Generate. A VLM (Gemini) performs affordance reasoning: "what would a person do here?" — returning descriptions like "someone sitting on the sofa watching TV." An image editing model then *generates* (not pastes) humans matching the scene's lighting, perspective, and style. Results are scene-adaptive: the same "watching TV" action adjusts pose and orientation to the specific sofa and TV. These plausibilities emerge from visual priors rather than complex hand-crafted constraints.
Lift. An optimization procedure (inspired by PhySIC) lifts 2D results to 3D: 1. Monocular depth estimation (MoGe) back-projected to align with scene geometry, initializing the human. 2. Optimize SMPL-X parameters (pose θ, shape β, translation t, global scale s) with losses: projection alignment L_proj, penetration penalty L_pen, PhySIC-based contact loss L_contact, pose prior L_prior.
1.3 Quality Control and Scale
Four filtering layers: depth-boundary filtering (editing model altered scene geometry), human-size filtering (1.2m–2.2m), penetration filtering, and contact filtering (DECO estimator). ~80% of samples survive. Fully automated over HM3D, yielding the InHabitants dataset: 78,000+ samples, ~800 building-scale scenes, single- and multi-person interactions.
1.4 Experiments
1.5 The Essence
InHabit is knowledge distillation — from a 2D foundation model into a 3D dataset. Semantic richness from VLM commonsense, visual realism from image generation, geometric precision from optimization. Limitation: it produces static poses, not dynamic motion; the authors note extension to video and dynamic scenes is future work.
Chapter 2: CoInteract — Fixing Broken Hands
2.1 Structural Collapse in Video Generation
Current video models (Wan, HunyuanVideo, SkyReels) produce beautiful faces and fluid motion, but hands fail — extra fingers, penetration, floating objects. The problem isn't model size but weak supervision in RGB space: geometric constraints (where fingers bend around a cup, where contact boundaries lie) are implicit and confusable in pixel color.
Two fragile spots: (1) structural stability of sensitive regions (hands/faces — fine detail, small pixel area), (2) physically plausible contact with no direct supervision.
2.2 Human-Aware MoE
A Mixture-of-Experts router trained with spatial supervision from MediaPipe/Face Mesh bounding boxes: face tokens go to a face expert, hand tokens to a hand expert, everything else to generalists. Experts are lightweight — near-zero parameter overhead. Routing heatmaps cleanly separate hand/face tokens even during fast motion.
2.3 Spatially-Structured Co-Generation
Both streams share the DiT backbone with independent patch embeddings and per-stream adaptive layer normalization.
3D RoPE: streams concatenated along the width axis (RGB at w ∈ [0, W], HOI at w ∈ [-W, 0], shared height/time indices); timeline encodes history frames (t < 0), generation window t ∈ [0, T], and reference image (t ≫ T).
Two-stage training:
The payoff: at inference the HOI branch is simply removed with no overhead, while backbone weights retain the interaction-geometry priors learned via HOI←RGB cross-attention gradients.
2.4 Data Engineering
Qwen-Edit decouples person and object into reference images; a validation module filters bad triplets; SAM3 segments objects, SAM3D-body recovers body meshes; mesh projection + object masks form the HOI stream.
2.5 Results
On unseen objects, AnchorCrafter shows identity drift and unnatural interaction boundaries; InteractAvatar degrades over long sequences. CoInteract maintains physically plausible interactions and structural stability throughout, improving structure stability, logical consistency, and interaction realism.
2.6 The Essence
CoInteract compensates for RGB models' geometric blindness via structured supervision — an implicit constraint approach: no per-sample optimization, just enough "correct geometry" during training for the model to internalize it.
Chapter 3: Head-to-Head
Chapter 4: Feynman-Style Scrutiny
Plain-language versions
InHabit: You have an empty house. You ask someone who has seen countless photos (VLM) what people should be doing there; a skilled painter (image model) draws it; you "lift" the drawn person into 3D, ensuring feet touch the floor and no sofa penetration. Repeat tens of thousands of times.
CoInteract: Video models break hands, so you train the model to draw video *and* skeleton simultaneously. The skeleton branch teaches correct geometry, then is discarded at generation time.
Cargo cult checks
First-principles view
"Looking right" requires physical, semantic, visual, and temporal correctness. Each paper covers only a subset. A possible fusion: InHabit-style physically precise 3D poses as motion conditions for CoInteract-style video synthesis (perhaps a triple-stream RGB + HOI + 3D pose model); or a data flywheel where generated HOI videos feed back into 3D data generation.
The boundary of "understanding"
Neither system understands human interaction. InHabit's VLM doesn't know what "sitting" means — only statistical positions of human pixels near sofas. CoInteract's DiT doesn't know what "grasping" means — only latent patterns that look like grasping. A model that truly understood would answer: "Can a person sit on an overturned sofa? On a sofa covered in spikes?" These demand causal reasoning. Practically, it doesn't matter — but scientifically, mistaking statistical correlation for understanding risks missing the real breakthroughs. They are simply very good at imitation.
Conclusion: Two Roads, One Destination
InHabit: build the data first, then train models — scale of 78,000 samples across 800 scenes. CoInteract: generate content directly — quality via structured supervision at zero inference cost. Both converge on the same goal: making AI truly understand how humans move in the physical world.
If you work on embodied AI, watch InHabit — its dataset may boost contact estimation. If you do content generation, watch CoInteract — dual-stream training may be the fix for broken hands. And if you think about whether AI can truly understand human interaction, read both — not because they solved it, but because they honestly show the current boundary and the effort to push it.
That is how it is.