Illusion3D: Generating 3D Illusions using Diffusion Models

Abstract

Recent works like [1] and [2] have explore the possibility of generating 2D illusion image using diffusion based method. But 3D illusion remain unexplored. We investigate the problem of texturing and modeling three-dimensional objects to create illusions. Using a differentiable renderer, an image of an object, such as a cube, may be rendered from several camera positions. We use optimization based method like in [2] to train the texture map of a 3D object and generate illusion with given prompts and camera view. This approach can be extended to various other transformations of the scene, such as by using several configurations of a Rubik’s cube as views, or by viewing several reflective cylinders over a flat texture. We provide quantitative and qualitative assessments of the quality of our illusions through thorough experimentation with a variety of prompts and views. Additionally, we illustrate the effectiveness of our illusions in the real world.