|dc.description.abstract||With recent advances in robotics, general purpose robots like Baxter are
quickly becoming a reality. As robots begin to collaborate with humans in everyday
workspaces, they will need to understand the functions of objects and their
parts. To cut an apple or hammer a nail, robots need to not just know a tool’s name,
but they must find its parts and identify their potential functions, or affordances.
As Gibson remarked, “If you know what can be done with a[n] object, what it can
be used for, you can call it whatever you please.”
We hypothesize that the geometry of a part is closely related to its affordance,
since its geometric properties govern the possible physical interactions with the environment.
In the first part of this thesis, we investigate how the affordances of tool
parts can be predicted using geometric features from RGB-D sensors like Kinect.
We develop several approaches to learn affordance from geometric features: using
superpixel based hierarchical sparse coding, structured random forests, and convolutional
neural networks. To evaluate the proposed methods, we construct a large
RGB-D dataset where parts are labeled with multiple affordances. Experiments
over sequences containing clutter, occlusions, and viewpoint changes show that the
approaches provide precise predictions that can be used in robotics applications.
In addition to geometry, the material properties of a part also determine its
potential functions. In the second part of this thesis, we investigate how material
cues can be integrated into a deep learning framework for affordance prediction. We
propose a modular approach for combining high-level material information, or other
mid-level cues, in order to improve affordance predictions. We present experiments
which demonstrate the efficacy of our approach on an expanded RGB-D dataset,
which includes data from non-tool objects and multiple depth sensors. The work
presented in this thesis lays a foundation for the development of robots which can
predict the potential functions of tool parts, and provides a basis for higher level
reasoning about affordance.||en_US