Object-Attribute Compositionality for Visual Understanding

dc.contributor.advisorShrivastava, Abhinav Dren_US
dc.contributor.authorSaini, Niraten_US
dc.contributor.departmentComputer Scienceen_US
dc.contributor.publisherDigital Repository at the University of Marylanden_US
dc.contributor.publisherUniversity of Maryland (College Park, Md.)en_US
dc.date.accessioned2024-09-23T06:27:25Z
dc.date.available2024-09-23T06:27:25Z
dc.date.issued2024en_US
dc.description.abstractObject appearances evolve overtime, which results in visually discernible changes in their colors, shapes, sizes and materials. Humans are innately good at recognizing and understanding the evolution of object states, which is also crucial for visual understanding across images and videos. However, current vision models still struggle to capture and account for these subtle changes to recognize the objects and underlying action causing the changes. This thesis focuses on using compositional learning for recognition and generation of attribute-object pairs. In the first part, we propose to disentangle visual features for object and attributes, to generalize recognition for novel object-attribute pairs. Next, we extend this approach to learn entirely unseen attribute-object pairs, by using semantic language priors, label smoothing and propagation techniques. Further, we use object states for action recognition in videos where subtle changes in object attributes and affordances help in identifying state-modifying and context-transforming actions. All of these methods for decomposing and composing objects and states generalize to unseen pairs and out-of-domain datasets for various compositional zero-shot learning and action recognition tasks. In the second part, we propose a new benchmark suite Chop \& Learn for a novel task of Compositional Image Generation as well as discuss the implications of these approaches for other compositional tasks in images, videos, and beyond. We further extend insertion and editing of attributes of objects consistently across frames of videos, using off-the-shelf training free architecture and discuss the future challenges and opportunities of compositionality for visual understanding.en_US
dc.identifierhttps://doi.org/10.13016/anle-scci
dc.identifier.urihttp://hdl.handle.net/1903/33459
dc.language.isoenen_US
dc.subject.pqcontrolledComputer scienceen_US
dc.subject.pquncontrolledCompositional Learningen_US
dc.subject.pquncontrolledComputer Visionen_US
dc.subject.pquncontrolledMachine Learningen_US
dc.titleObject-Attribute Compositionality for Visual Understandingen_US
dc.typeDissertationen_US

Files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
Saini_umd_0117E_24649.pdf
Size:
103.72 MB
Format:
Adobe Portable Document Format