Kozminski Techblog

In 2020, almost everyone encountered the problem of face recognition when unlocking a phone, because of wearing a mask.
And I will delight you, there is a man who was able to solve this problem.
Cheng Zhang and his colleagues developed the tool, called C-Face at Cornell University in Ithaca, New York.

C-Face (Contour-Face) is an ear-mounted wearable sensing technology that uses two mini-cameras to continuously reconstruct facial expressions by deep learning contours of the face. When facial muscles move, the contours of the face change from the point of view of the ear-mounted cameras.
“With previous technology to reconstruct facial expression, you had to put a camera in front of you. But that brings a lot of limitations,” says Zhang. “Right now, many people are wearing a face mask, and standard facial tracking will not work. Our technology still works because it doesn’t rely on what your face looks like.”

“The most exciting finding is that facial contours are highly informative of facial expressions,” the researchers wrote. It’s true , because our facial muscles expand and contract while we conduct a facial expression. They push and pull the skin and control the tension of the facial muscles that are nearby. This influence causes the cheeks’ shape (contours) to shift from the ear’s point of view.
The model translates the images of cheeks to 42 points of the facial feature, or landmarks, reflecting the shapes and locations of the mouth , eyes, and eyebrows, as changes in speech are the most influenced by those features.

Zhang and his colleagues tested C-Face in nine volunteers. When compared with a library of images taken with a front-positioned camera, the device accurately predicted facial arrangements for different expressions.

C-Face is still at an early stage and there are technological challenges that need to be addressed before it can be rolled out more generally, such as battery life. Zhang also admits that it may be an issue to guarantee user privacy. “There’s so much private information we can get from this device,” he says. “I think that’s definitely one issue we need to address.”

Sources :

https://ad164934-109c-47d8-a0fd-ee214ece47db.filesusr.com/ugd/2311aa_832c9241fdd74d7296976cefeccb43b4.pdf

https://www.scifilab.org/c-face

https://news.cornell.edu/stories/2020/10/earphone-tracks-facial-expressions-even-face-mask