Visual and Auditory Scene Parsing
, CSAIL, Computer Vision Group
Date: Thursday, May 09, 2019
Time: 1:00 PM to 2:00 PM
Location: Hewlett Room 32-G882
Event Type: Thesis Defense
Host: Antonio Torralba
Contact: Fern Keniston, firstname.lastname@example.org
Speaker URL: None
TALK: Thesis Defense: Visual and Auditory Scene Parsing
Scene parsing is a fundamental topic in computer vision and computational audition, where people develop computational approaches to achieve human perceptual system's ability in understanding scenes, e.g. group visual regions of an image into objects and segregate sound components in a noisy environment. This thesis investigates fully-supervised and self-supervised machine learning approaches to parse visual and auditory signals, including images, videos, and audios.
Visual scene parsing refers to densely grouping and labeling of image regions into object concepts. First I build MIT scene parsing benchmark based on a large scale, densely annotated dataset ADE20K. This benchmark, together with the state-of-the-art models we open source, offers a powerful tool for the research community to solve semantic and instance segmentation tasks. Then I investigate the challenge of parsing a large number of object categories in the wild. An open vocabulary scene parsing model which combines convolutional neural network with structured knowledge graph is proposed to address the challenge.
Auditory scene parsing refers to recognizing and decomposing sound components in complex auditory environments. I propose a general audio-visual self-supervised learning framework that learns from a large amount of unlabeled internet videos. The learning process discovers the natural synchronization of vision and sounds without human annotation. The learned model achieves the capability to localize sound sources in videos and separate them from mixture. Furthermore, I show that motion cues in videos are tightly associated with sounds, which help in solving sound localization and separation problems.
AI & Machine Learning, Graphics & Vision
Created by Fern D Keniston at Thursday, May 02, 2019 at 6:37 PM.