Modality-Agnostic Representation Learning via Hierarchical Variational Auto-Encoders

Speaker: Reuben Dorent , Harvard Medical School

Date: Wednesday, November 15, 2023

Time: 2:00 PM to 3:00 PM Note: all times are in the Eastern Time Zone

Public: Yes

Location: 32-D451

Event Type: Seminar

Room Description:

Host: Polina Golland, CSAIL

Contact: Polina Golland, polina@csail.mit.edu

Relevant URL:

Speaker URL: None

Speaker Photo:
None

Reminders to: seminars@csail.mit.edu

Reminder Subject: TALK: Modality-Agnostic Representation Learning via Hierarchical Variational Auto-Encoders

Learning pixel-level modality-agnostic representation of multi-modal imaging data is a challenging and open problem. In this work, Dr. Reuben Dorent will introduce MHVAE, a deep hierarchical variational auto-encoder (VAE) that generates modality-agnostic representations at the pixel level and allows for missing imaging modalities at training and testing time. Extending multi-modal VAEs with a hierarchical latent structure, a parametrization of the approximate posterior is introduced with a factorization similar to the true posterior, which can be expressed as a combination of unimodal variational posteriors. A simple optimization strategy is proposed to encourage learned representations to be modality-agnostic. Experiments on a database of intra-operative ultrasound (iUS) and Magnetic Resonance (MR) images demonstrate the effectiveness of the proposed approach at generating pixel-level representations that retain the content information while being similar for different sets of input modalities.

Research Areas:

Impact Areas:

See other events that are part of the Biomedical Imaging and Analysis 2023 - 2024.

Created by Polina Golland at Monday, November 06, 2023 at 10:35 PM.