This article reflects information as of 2020. For the latest details, please contact us.
Written by: Tomohiro Koizumi, Representative Director, tentus inc.
Have you ever played with apps that use face photos?
Apps that let you 【change the gender】 of your face, 【add items】 like cat ears, or 【change the image】 such as the background — I've played with them a few times myself.
(Converting my photo from male → female with FaceApp)
This feature uses face-recognition technology, and it's one of the technologies that benefited most from the advance of AI.
Face-recognition technology is already in practical use in surveillance cameras and the like, so it's grown to a level where it can produce quite high accuracy even in casual mobile content.
Face-Recognition Technology in 2011
At first it was a common, ordinary planning consultation.
"Can we do something fun using a celebrity's face photo?"
I think I only had about three days from receiving the request to submitting the plans, and since no budget was presented in particular, I submitted several plans, large and small.
【1. Small budget】 A digital fukuwarai using facial parts
【2. Medium budget】 An aging simulation using morphing technology
【3. Large budget】 Composite content using face photos
I believe it was something like that.
In the end, the planner liked plan 3, and we decided to make composite content using face photos.
Here are the details of the plan.
Baby-Generation Content Using Photos of the Celebrity and the User
Yes — just putting it into words feels sketchy.
- Choose your favorite celebrity 2. Upload your own face photo to the server 3. The two's child is generated
That was the content, but back then face-recognition technology wasn't well developed and its ability to detect the outlines of facial parts from a face photo — to set facial keypoints — was very low, so we sorted the face photos by similarity to preset per-part features.
"This person's eyebrows are thin and shaped like a hachi (八), so the eyebrow pattern is pattern A-23"
Like that, we sorted each part with the face-recognition engine.
By doing this, even if the face-recognition engine couldn't properly capture the features, we could sort it into a close-pattern part, enabling the generation of images that were "not alike" but "kind of plausible."
Further, at random:
Eyebrows from the celebrity, nose from the user, outline from the celebrity, lips from the celebrity
We internally decided at random which to resemble, and further decided on a boy version and a girl version to generate — I forget the exact number of digits, but it made possible tens of millions of variations.
And the Image That Came Out Was…
It splendidly fell straight to the bottom of the uncanny valley.
What is the uncanny valley?
Proposed by robotics engineer Masahiro Mori in 1970. Regarding humans' emotional reactions to robots, Mori predicted that as a robot is made more human-like in appearance and movement, reactions become more favorable and empathetic — but at a certain point they suddenly turn into strong revulsion. He thought that once appearance and movement become indistinguishable from a human's, it again turns to stronger favorability, and one comes to feel a familiarity as with a human. One can predict a difference in the viewer's emotional reaction between a robot "extremely close to human" and one "exactly the same as human" in appearance and movement. The valley of strong revulsion that appears when graphing the difference between these two emotional reactions is called the "uncanny valley." For humans and robots to collaborate productively, it's essential that humans can feel familiarity toward robots, but a "near-human" robot is felt to be terribly "strange" by humans, and, being unable to feel familiarity, it was named after that. Excerpted from Wikipedia.
In the end, because we only used face recognition for sorting parts, we couldn't perform per-part optimization or overall optimization at generation time, as today's AI-based face recognition does — so what was generated was "an image that at a glance looks like a baby, yet is very scary."
With illustrations, the eeriness might have been diluted somewhat, but I think the decisive point that dropped it into the uncanny valley was breaking down actual baby images part by part to generate the image.
At the time, that eeriness, combined with everything else, made it content that got a fair amount of buzz — but if you tried to build the same mechanism today, you could do it really easily using the latest face-recognition engine.
I'll wrap up this case with the tidy conclusion that 【technological progress involves a lot of trial and error】.
Yep.