- Towards Robust and Practica...
- Edit Event
- Cancel Event
- Preview Reminder
- Send Reminder
- Other events happening in October 2023
Towards Robust and Practical Neural Video Conferencing
Speaker:
Vibhaalakshmi Sivaraman
, MIT CSAIL
Date: Monday, October 16, 2023
Time: 10:00 AM to 11:00 AM Note: all times are in the Eastern Time Zone
Public: Yes
Location: Seminar Room D463 (Star)
Event Type: Thesis Defense
Room Description:
Host: Mohammad Alizadeh
Contact: Vibhaalakshmi Sivaraman, vibhaa@csail.mit.edu
Relevant URL:
Speaker URL: https://people.csail.mit.edu/vibhaa/
Speaker Photo:
None
Reminders to:
seminars@csail.mit.edu, g9@csail.mit.edu
Reminder Subject:
TALK: Monday 10-16-2023 Thesis Defense: "Towards Robust and Practical Neural Video Conferencing"
Abstract: Video conferencing systems suffer from poor user experience when network conditions deteriorate because current video codecs simply cannot operate at extremely low bitrates. Recently, several neural alternatives have been proposed that reconstruct talking head videos at very low bitrates using sparse representations of each frame such as facial landmark information. However, these approaches produce poor reconstructions in scenarios with major movement or occlusions over the course of a call, and do not scale to higher resolutions. This thesis first presents Gemino, a new neural compression system for video conferencing based on a novel high-frequency-conditional super-resolution pipeline. Gemino upsamples a very low-resolution version of each target frame while enhancing high-frequency details (e.g., skin texture, hair, etc.) based on information extracted from a single high-resolution reference image. We use a multi-scale architecture that runs different components of the model at different resolutions, allowing it to scale to resolutions comparable to 720p, and we personalize the model to learn specific details of each person, achieving much better fidelity at low bitrates. We implement Gemino atop aiortc, an open-source Python implementation of WebRTC, and show that it operates on 1024x1024 videos in real-time on a Titan X GPU, and achieves 2.2–5x lower bitrate than traditional video codecs for the same perceptual quality. The thesis will also cover alternate designs for neural video conferencing solutions that leverage attention techniques instead of optical flow, along with neural compression techniques for loss-resilience.
Research Areas:
Systems & Networking
Impact Areas:
Created by Vibhaalakshmi Sivaraman at Thursday, September 28, 2023 at 3:15 PM.