Mona Lisa: Smiling? Computer Scientists Develop Software That Evaluates Facial Expressions

August 1, 2006 — Computer vision software can now map a person's face onto a mesh computer model and calculate facial expressions based on facial points such as lip curvature, eyebrow position, and cheek contraction. The software detects happiness, disgust, fear, anger, surprise and sadness with 85 percent accuracy, but researchers don't yet have the technology to detect more subtle emotions.

AMSTERDAM, Netherlands -- Computers are becoming more and more like humans! Now, new technology allows them to detect what we're feeling!

You've heard the good and the bad about the Da Vinci Code. Now, cutting-edge technology helps unleash another Da Vinci mystery -- what the famous Mona Lisa was feeling.

Have any guesses?

Actually, the Mona Lisa's expression is 83-percent happy, 9-percent disgusted, 6-percent fearful, and two-percent angry.

A computer vision expert in Amsterdam used software developed at the University of Illinois at Urbana-Champaign to design emotion recognition technology. With a Web cam, they map a person's face onto a mesh computer model. It calculates the expression based on facial points like lip curvature, eyebrow position, and cheek contraction -- with 85-percent accuracy.

"That's pretty good, actually, because it's very close to what the human can do," computer scientist Nicu Sebe, of the University of Amsterdam, in the Netherlands, tells DBIS. The software detects happiness, disgust, fear, anger, surprise and sadness, but researchers don't yet have the technology to detect more subtle emotions.

The researchers found George Bush was feeling surprise, fear and sadness during a speech regarding the war in Iraq. Michael Jackson was 33-percent fearful in his mug shot and angry and disgusted as the press snapped pictures after his trial.

Emotion-recognition technology may be used to detect that a driver is getting sleepy at the wheel and have an alert signal and to detect how you feel about certain items while you're shopping ... Proof it takes a look at the past to pave the way for the future.

BACKGROUND: For centuries, art lovers have wondered about the Mona Lisa's mysterious smile, and what Mona Lisa may have been thinking or feeling the time she was painted. Now, scientists in The Netherlands have used emotion-recognition software to determine the Mona Lisa's emotions while sitting for her portrait by Leonardo da Vinci. Other applications of emotion recognition software might be to detect terror suspects on the basis of their emotions, not just on their physical characteristics.

HOW IT WORKS: The software uses a network of points over a 3D model of the face of Leonardo's most famous subject. This looks at features such as the curvature of the lips and crinkles around the eyes to give a score for six basic emotions. The software scored the Mona Lisa for happiness, disgust, fear and anger, but found no evidence of surprise or sadness. The exact breakdown of the Mona Lisa's emotions, according to the software program, were 83 percent happy, 9 percent disgusted, 6 percent fearful, and 2 percent angry.

IN DIVINE PROPORTION: Fans of The Da Vinci Code know that the Mona Lisa smile isn't the only mystery associated with Leonardo's masterpiece. In 1509, he collaborated with mathematician Fra Luca Pacioli and artist Piero della Francesca on a book about the golden ratio: If a line is divided into two unequal lengths, in such a way that the ratio of the longer segment to the shorter segment will be the same as the ratio of the whole line to the longer segment, the resulting number will be something close to 1.618. Some art historians say that within the painting, the relationship between the Mona Lisa's right shoulder and cheek and her left shoulder and cheek forms a golden triangle whose shortest sides are in divine proportion to its base.

ROBOTS THAT FEEL? Emotion-recognition software is part of the field of human-computer interaction, a technology that allows computers to respond to human beings appropriately by reading the expression on their faces. The most famous example is a robot called Kismet, built by researchers at MIT, that reacts to tone of voice and visual cues such as body movement, posture, and facial expression in order to interact socially with human beings. The robot does not experience emotions, but its creators are hoping to create a convincing simulation as Kismet continues to learn.

The Institute of Electrical and Electronics Engineers, Inc., contributed to the information contained in the TV portion of this report.

Note: This story and accompanying video were originally produced for the American Institute of Physics series Discoveries and Breakthroughs in Science by Ivanhoe Broadcast News and are protected by copyright law. All rights reserved.