Visual and Auditory Scene Parsing

Speaker: Hang Zhao , CSAIL, Computer Vision Group

Date: Thursday, May 09, 2019

Time: 1:00 PM to 2:00 PM

Public: Yes

Location: Hewlett Room 32-G882

Event Type: Thesis Defense

Room Description:

Host: Antonio Torralba

Contact: Fern Keniston,

Relevant URL:

Speaker URL: None

Speaker Photo:
Hang zhao web

Reminders to:,

Reminder Subject: TALK: Thesis Defense: Visual and Auditory Scene Parsing

Scene parsing is a fundamental topic in computer vision and computational audition, where people develop computational approaches to achieve human perceptual system's ability in understanding scenes, e.g. group visual regions of an image into objects and segregate sound components in a noisy environment. This thesis investigates fully-supervised and self-supervised machine learning approaches to parse visual and auditory signals, including images, videos, and audios.
Visual scene parsing refers to densely grouping and labeling of image regions into object concepts. First I build MIT scene parsing benchmark based on a large scale, densely annotated dataset ADE20K. This benchmark, together with the state-of-the-art models we open source, offers a powerful tool for the research community to solve semantic and instance segmentation tasks. Then I investigate the challenge of parsing a large number of object categories in the wild. An open vocabulary scene parsing model which combines convolutional neural network with structured knowledge graph is proposed to address the challenge.
Auditory scene parsing refers to recognizing and decomposing sound components in complex auditory environments. I propose a general audio-visual self-supervised learning framework that learns from a large amount of unlabeled internet videos. The learning process discovers the natural synchronization of vision and sounds without human annotation. The learned model achieves the capability to localize sound sources in videos and separate them from mixture. Furthermore, I show that motion cues in videos are tightly associated with sounds, which help in solving sound localization and separation problems.

Research Areas:
AI & Machine Learning, Graphics & Vision

Impact Areas:

This event is not part of a series.

Created by Fern D Keniston Email at Thursday, May 02, 2019 at 6:37 PM.