JanusMesh Explained: One Sculpture, Two Worlds — Fast 3D Visual Illusion Generation

Forum topic · 小凯 · 2026-06-21

Summary

JanusMesh is a training-free method from a four-person team at National Yang Ming Chiao Tung University that generates 3D visual illusion meshes — single 3D objects that look like one object (e.g., a peacock) from angle A and a completely different object (e.g., a pineapple) from angle B. Previous approaches either used slow SDS-based optimization (~40 minutes, with oversaturated colors) or direct concatenation of two models, which produced visible seams and semantic leakage. JanusMesh builds on TRELLIS with a two-stage pipeline: (1) dual-branch cross-space denoising, where two text-conditioned branches share the same noise, decode to voxel space, align via CLIP-guided rotation, and blend through signed distance field (SDF) averaging at each of 25 denoising steps; and (2) view-conditioned texture synthesis using Stable Diffusion with projection and mesh texture aggregation. A Noise Guidance mechanism enables three-semantic illusions at 0°/120°/240° viewpoints. Generation takes only 3–5 minutes on a single GPU and requires no task-specific training. Evaluation uses CLIP similarity, GPT-4.1-mini visual judgment, FID/KID, and object detection metrics.

JanusMesh: When One Sculpture Hides Two Worlds

*A deep dive into JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising (arXiv:2606.20563, accepted to ECCV 2026), from a four-person team at National Yang Ming Chiao Tung University.*

The Problem: 3D Visual Illusion

A 3D visual illusion is a single 3D object that looks like object X from viewpoint A, object Y from viewpoint B, and an abstract, unrecognizable shape from any other angle — like the Roman god Janus, facing two directions at once.

Two prior approaches existed, both flawed:

SDS-based optimization (DreamFusion, Shape From Semantics): iteratively optimize a 3D representation with Score Distillation Sampling. Problems: extremely slow (~40 minutes per object) and severe color oversaturation, as SDS keeps pushing saturation upward.
Direct Concatenation: generate two 3D models separately and stitch them together. Problems: visible geometric seams and semantic leakage — fragments of one object bleeding into the other's designated viewpoint.

The Method: Cross-Space Denoising

JanusMesh is training-free and built on TRELLIS (CVPR 2025), a 3D generator using Rectified Flow with a two-stage structure: sparse voxel structure prediction, then appearance/texture refinement. JanusMesh intervenes only in the first stage, since geometry fusion must happen at the skeleton level.

Stage 1: Dual-Branch Denoising

At each of 25 denoising steps:

1. Branch A ("a peacock") and Branch B ("a pineapple") each denoise the same shared noise latent — sharing the input noise is key to avoiding two independent objects. 2. Both predicted latents are decoded into voxel space. 3. CLIP-guided orientation alignment searches for the best rotation so the two shapes align correctly in space. 4. SDF blending: the aligned voxel fields are merged by averaging their signed distance fields, producing a smooth, gradient transition instead of a boolean-union seam. 5. The blended result is re-encoded back to latent space for the next step.

This latent → voxel → latent shuttling is the "cross-space" in the name — a trick borrowed from LookingGlass, since direct latent-space geometric operations cause distribution shift. The paper states: *"This cross-space denoising ensures the geometric validity of the blended mesh."*

Noise Guidance for three semantics: extending to three objects (e.g., grapes at 0°, pineapple at 120°, bamboo at 240°) increases geometric conflict. A Space Control Guidance mechanism, using a pre-generated single-semantics prior, stays active for the first 20 of 25 steps.

Stage 2: View-Conditioned Texture Synthesis

With geometry fixed, textures are painted using Stable Diffusion:

1. Generate target-view images for each semantic (peacock view, pineapple view). 2. Un-project the 2D images onto the 3D mesh surface. 3. Mesh Texture Aggregation merges them: regions visible from view θ₁ get peacock texture, regions visible from θ₂ get pineapple texture, invisible regions get neutral textures to prevent leakage, with smooth blending at boundaries.

Geometry and texture are deliberately decoupled to reduce search-space complexity, guarantee geometric consistency, and keep the pipeline modular.

Results

| Method | Time | Geometry | Semantics | Color | |---|---|---|---|---| | Shape From Semantics (SDS) | ~40 min | Medium | Medium | Poor (oversaturated) | | Direct Concatenation | Fast | Poor (seams) | Poor (leakage) | Medium | | JanusMesh | 3–5 min | Excellent | Excellent | Excellent |

An order-of-magnitude speedup over SDS baselines, with higher quality across the board.

Evaluation protocol (four layers):

CLIP similarity between rendered target views and text prompts
GPT-4.1-mini judgment — whether the model correctly identifies the rendered object
FID / KID distributional metrics
Object detection (e.g., YOLO) as a hard binary success criterion

Ablations confirm every component matters: fusing directly in latent space (without the voxel-space detour) degrades geometry; plain averaging underperforms SDF blending; skipping CLIP alignment causes misaligned fusion; global (non-view-conditioned) texturing causes semantic leakage from non-target views.

Three-object illusions work qualitatively, though with heavier geometric conflict requiring stronger Noise Guidance — an honestly reported limitation.

Why It Matters

3–5 minutes vs. 40 minutes, on a single GPU, fully zero-shot — no dataset of illusion sculptures needed.
Applications span art installations, product design (a vase that reads differently from sofa vs. dining table), game assets, and architectural facades.
Philosophically, a 3D illusion object is a macroscopic nod to reference-frame dependence: the same entity carries multiple incompatible truths, resolved only by where you stand.

> "Turn it one way, it's a peacock. Turn it the other, it's a pineapple. The blurred angles in between? Those are left to the imagination."

References

1. Zhang, S.-L., Cheng, H.-H., Yang, T.-J., & Liu, Y.-L. (2026). *JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoising*. arXiv:2606.20563. ECCV 2026. Project Page 2. Xiang, J., et al. (2025). Structured 3D Latents for Scalable and Versatile 3D Generation (TRELLIS). CVPR, 21469–21480. 3. Poole, B., et al. (2022). DreamFusion: Text-to-3D Using 2D Diffusion. arXiv:2209.14988. 4. Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 10684–10695. 5. Radford, A., et al. (2021). Learning Transferable Visual Models from Natural Language Supervision. ICML, 8748–8763. 6. Geng, Z., et al. (2024). LookingGlass: Generative Anaglyphs from Any Image. arXiv:2404.06448. 7. Geng, Z., Kwon, M., & Essa, I. (2024). Visual Anagrams. CVPR. 8. Li, Y., et al. (2024). SyncTweedies. arXiv:2409.08276. 9. Oliva, A., Torralba, A., & Schyns, P. G. (2006). Hybrid Images. ACM TOG, 25(3), 527–532. 10. Lipman, Y., et al. (2023). Flow Matching for Generative Modeling. ICLR.