Learning from Hidden Signals in Visual Data

Abstract

Cameras capture rich signals from the physical world, but extracting them requires incorporating additional priors. The work in this thesis draws on two sources of priors: known physical and geometric facts, and generative priors learned from large-scale data. In the first part, the structure comes from physics and geometry. We use this structure to extract information from reflections, motion, and defocus cues in photos and videos. We reconstruct 3D scenes from the reflections in a person's eyes, using the known geometric prior on the human cornea. We magnify subtle motions in handheld videos, by performing motion magnification in 3D, which naturally separates scene motion from camera motion. We enable post-capture defocus control on smartphones using the dual-camera setup. We use the thin-lens model combined with stereo depth to map the focus plane to a defocus map, and train a model to match any target defocus map by blending the two cameras' outputs. In the second part, the structure comes from generative models. We use them to enable image editing, 3D reconstruction under inconsistent lighting, consistent multi-view editing, and simulation of robot interactions. We use video frame pairs as natural supervision for photorealistic image editing, and build a system that takes a coarse user edit and makes it realistic. Beyond 2D images, we train a multi-view relighting model that harmonizes a collection of photos taken under inconsistent illumination to a single consistent lighting. Then, we use the harmonized photos to reconstruct the object in 3D while preserving specular highlights and reflections. For generalized multi-view editing, we show that two pretrained diffusion models, one for 2D editing and one for multi-view generation, can be composed at inference by steering their sampling trajectories, enabling consistent multi-view editing for relighting, stylization, and spatial editing. Finally, we use the generative prior of video models to simulate robot interactions, expressing actions as partially revealed motion in the video itself. Depending on the motion revealed, the model can either act as a simulator that visualizes the effects of different robot policies, or synthesize the robot motion that achieves a desired object motion. The generality of this action representation yields a unified world model in a scalable and embodiment-agnostic setting.