Toward Adaptive and Efficient Synthesis of Appearance, Dynamics and Semantics

Abstract

Generative modeling for visual content has advanced rapidly, enabling both the synthesis of novel images and videos and the transformation of existing visual assets. Among recent approaches, diffusion-based methods have played a major role due to their high-fidelity generation quality, robust scalability, and broad generalization across tasks. Despite these advances, modern large-scale generative models remain limited in adaptability and computationally expensive. This thesis investigates adaptive and efficient methods for image editing and video generation.

We begin with content-adaptive image color editing. Color adjustment is inherently content- and region-dependent, whereas generic image translation models often fail to produce automatic yet aesthetically appropriate results. To address this, we develop an image-to-image diffusion model trained with auxiliary color restoration tasks, enabling the model to capture chromatic preferences from raw content and perform adaptive color editing without manual tuning.

We then introduce one-shot video motion customization, a task that adapts a pre-trained text-to-video diffusion model from a single reference video to synthesize its motion with novel subjects and scenes. This setting addresses a gap between the weak motion controllability of standard text-conditioned video generation and the rigidity of prior editing-based approaches. By injecting temporal signals from the reference video, our method enables flexible control over motion intensity, viewpoint, and subject count. We further analyze spatiotemporal disentanglement across denoising timesteps, showing that motion customization can be substantially simplified without additional modules or training stages.

We next study efficient video generation through implicit neural representations. Conventional video latent diffusion models typically inherit frame-wise tokenization from image models, overlooking the coherence of video as a holistic spatiotemporal signal. This leads to redundant representations, growing reliance on cross-frame attention for temporal consistency, and poor scaling as data resolution increases. To address these limitations, we explore implicit latent video generation based on instance-specific neural representations, shifting from frame-wise latent feature maps to compact functional representations of entire video clips. This perspective enables more holistic video synthesis and a substantially lighter decoding process, pointing to a more efficient alternative to conventional latent video generation pipelines.

Finally, we present a noise-free framework for instructional image editing based on latent flow matching. Despite substantial progress in image editing, most diffusion- and flow-based methods remain noise-centric: they perturb the source image and reconstruct the target along a detoured trajectory, limiting both precision and efficiency. We reformulate instructional editing as direct source-to-target latent transport, repurposing pre-trained text-to-image diffusion models as source-anchored editors guided by concise instructions. This source-anchored formulation improves editing fidelity and inference efficiency, pointing toward a more direct treatment of instruction-guided visual transformation.