ModiFace to present AI video-processing breakthrough at CVPR

May 29, 2021

ModiFace has spent over a decade improving our understanding of objects in video. It is important for many of our applications, whether we are trying to locate a user’s lips, follow a strand of their hair, or correctly orient nail polish as a user waves their hand in front of their camera. Doing this on hundreds of frames to produce the best result in our virtual try-on trackers can be challenging. Our team set out to understand how machine learning algorithms could more easily and automatically track objects through space and time.

We introduce SSTVOS to tackle this problem. SSTVOS can trace each object in a scene forward and backward through time by predicting object "masks"—a silhouette of each object. SSTVOS looks at each pixel in a video and searches the rest of the video for similar pixels using a process called "attention". Attention produces a set of scores that indicate how similar each pixel is to other pixels in the video. SSTVOS then aggregates the resulting attention scores to predict each object's motion through time. SSTVOS is a product of an international collaboration between ModiFace, academia from the Toronto area (Vector Institute, University of Guelph), and France-based INSA Lyon.

This new technique will let us develop better, more accurate trackers that will power the future of our augmented reality. Tune in to CVPR 2021 as we present this groundbreaking work!