Grandma, what big eyes you have

Humans are a sentimental species.

That’s why an amazing service that animates old photographs to bring them to life was used more than a million times within 48 hours of being launched.

Photos of a loved one or long lost relative – even if they’re dog-eared and blurry in sepia or black and white – can be animated so the faces smile, blink and move again.
The service was developed by the genealogy site MyHeritage, which has helped 60 million people to trace their family history. Its users have uploaded millions of old photos, prompting its technicians to develop tools to colour them and improve their quality. Then they decided to inject some life to enhance the emotional connection. It found the solution by partnering with an Israeli artificial intelligence and deepfake company called D-ID. Deep Nostalgia was launched in February this year, and 50 million animations were created in just 18 days.

“We’ve always looks for special ways to allow people to connect with their ancestors, and one of the best ways to do that is through artificial intelligence,” said Gilad Japhet, the technology veteran and genealogist who founded MyHeritage. “It’s been very emotional to do this with my late father and see him smiling back at me. You get a ‘wow moment’ when you see a treasured family photo come to life with Deep Nostalgia,” he told CNN International.

D-ID has trained artificial intelligence algorithms by using ‘driver’ videos that show real humans moving, making facial expressions, and speaking. The AI replicates those movements in a still photo, turning it into a video where the photo smiles, blinks and matches any other movement in the driver video, including speaking.

MyHeritage recorded new driver videos using its own staff to capture sequences of small facial movements. The system automatically decides which sequence to apply to a photo based on its orientation.

Ideally it needs a high-resolution photo for the animation to work, but faces in historical photos tend to be small and blurry. So first it applies MyHeritage Photo Enhancer to increase the resolution and sharpen the focus. Then the animation is added almost instantly to create a short video that you can download and share.

To prevent people manipulating a photo of someone to have them say what they want, MyHeritage doesn’t include speech abilities, and hard-codes the photos to prevent further tampering.

How well does it work?

Most people have endless videos of their loved ones these days, but how much more poignant to have a favourite photograph that smiles back at you, recapturing a fleeting moment. I uploaded a photo of my late boyfriend Steve onto the Deep Nostalgia website. First it enhanced the image, then added the animation, which just took a few seconds.

When I clicked on the resulting image it hadn’t quite captured the way he moved or smiled, but it was realistic enough to flutter my heart and threaten a teardrop. If you do it with someone you don’t remember so well or never knew, like your long-dead grandma, it could be astonishing.

How do they do it?

The core technology is a branch of deep learning known as generative adversarial networks (GANs) invented by Ian Goodfellow in 2014 at the University of Montreal, one of the world’s top AI research institutes.
Before GANs, neural networks could classify existing content to understand speech or recognise faces, but couldn’t create new content. Now they can.
GANs use two separate neural networks, a generator and a discriminator, pitted against each other. Using a dataset of facial photos, the generator starts generating new images that are mathematically similar to the real images. The discriminator is fed photos from the original dataset and from the generator’s output, and has to identify which are synthetically generated.
As the generator tries to fool the discriminator, and the discriminator tries to detect the fake creations, they hone each other’s capabilities until the fakes become indistinguishable from the originals.

The fears around Deep Fakery

Human nature suggests that deepfake technology will inevitably be used for evil rather than for good.
Yes, some companies are using it to create training videos in multiple languages, and others are using it to enhance films. But by far the most common use is on porn sites, introducing the face of famous women or by amateurs making revenge porn featuring unwitting former partners. One app that was freely available before it was withdrawn was DeepNude, which converted images of clothed women into nudes.
Other risks are the spreading of political disinformation, or financial fraud, as deepfakes can deceive people, cause reputational damage, and undermine public trust in institutions.
“In the old days, if you wanted to threaten the United States, you needed 10 aircraft carriers and nuclear weapons and long-range missiles,” US Senator Marco Rubio said recently. “Today....all you need is the ability to produce a very realistic fake video that could undermine our elections, that could throw our country into tremendous crisis.”

First published in Brainstrom magazine.
If you want to test Deep Nostalgia for yourself, click here: