Hierarchical Scene Annotation

TitleHierarchical Scene Annotation
Publication TypeConference Paper
Year of Publication2013
AuthorsMaire, M., Yu S. X., & Perona P.
Other Numbers3620

We present a computer-assisted annotation system, together with a labeled dataset and benchmark suite, for evaluating an algorithm's ability to recover hierarchical scene structure. We evolve segmentation groundtruth from the two-dimensional image partition into a tree model that captures both occlusion and object-part relationships among possibly overlapping regions. Our tree model extends the segmentation problem to encompass object detection, object-part containment, and figure-ground ordering.

We mitigate the cost of providing richer groundtruth labeling through a new web-based annotation tool with an intuitive graphical interface for rearranging the region hierarchy. Using precomputed superpixels, our tool also guides creation of user-specified regions with pixel-perfect boundaries. Widespread adoption of this human-machine combination should make the inaccuracies of bounding box labeling a relic of the past.

Evaluating the state-of-the-art in fully automatic image segmentation reveals that it produces accurate two-dimension partitions, but does not respect groundtruth object-part structure. Our dataset and benchmark is the first to quantify these inadequacies. We illuminate recovery of rich scene structure as an important new goal for segmentation.


This work was partially supported by funding provided to ICSI through National Science Foundation CAREER award IIS : 1257700 (“Art and Vision: Scene Layout from Pictorial Cues”). Additional support was provided through ONR MURI N00014-10-1-0933 and ARO/JPL-NASA Stennis NAS7.03001. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors or originators and do not necessarily reflect the views of the National Science Foundation or any other funder.

Bibliographic Notes

Proceedings of the British Machine Vision Conference (BMVC), Bristol, United Kingdom

Abbreviated Authors

M. Maire, S. X. Yu, and P. Perona

ICSI Research Group


ICSI Publication Type

Article in conference proceedings