What is synthetic media, and how is it distinguished from digital human technology?

Synthetic media has become synonymous with the aspects of manipulation, particularly in deep fakes (and boy, will we come on to deep fakes). That has largely been down to how the technology has hit the public consciousness, driven by concerns over fake news and the genuine concern that people may use it to manipulate our perceptions of reality.

But synthetic media as a category and array of technologies is much more than just deep fakes. There are interesting, creative applications of synthetic media in popular culture and commercial settings.

There’s also an interesting intersection between synthetic media and digital humans, all of which we’ll unpack below.

What is synthetic media?

Synthetic media is an all-encompassing term for the artificial creation or modification of media by “machines” – particularly programs that rely on artificial intelligence and machine learning.

In other words, it’s media that is produced by technology. For this reason, you might also hear synthetic media called “AI-generated media”.

Some types of synthetic media today include AI-written music, text generation, imagery and video, voice synthesis and more. The field is ever expanding as synthetic media companies aim to disrupt more and more parts of traditional media, making new things easier to create.

For example, while recording a video featuring Tom Cruise can be an expensive and time consuming project, it’s now possible to create a deep-fake version of the actor with almost unnerving accuracy.

Which brings us on to perhaps the most popular and polarizing form of synthetic media today.

What about deep fakes?

No conversation about synthetic media can be made without including deep fakes – undoubtedly the most famous, infamous and synonymous form of synthetic media.

Deep fakes – a portmanteau of the phrases “deep learning” and “fake” – first emerged in late 2017. Powered by deep-learning technology called generative adversarial networks (GANs), deep fakes generally overlay the face of one person (usually a celebrity) onto the face of another person, often in real time.

It allows deep-fake practitioners to quickly and easily create media that manipulates what we see and hear.

Deep fake technology has exploded in popularity since 2017. In a strange parody of Moore’s Law, the number of deep fake videos published online doubles every six months, according to one estimate.

Clearly the nature of the technology poses many ethical concerns, from potential breaches of image rights to how it can be used to spread fake news or commit fraud. In the years since deep fakes emerged, they’ve been used to create celebrity versions of pornography without consent, spread fake news by making influential people say and do things they didn’t actually say or do, and even rewrite history.

On the more lighthearted side, deep fakes have been used to reunite Tupac and Snoop Dogg in a 2020 music video, make celebrity-endorsed adverts during social isolation and create memes – so many memes.

Meanwhile, the quality deep fakes can range from the impressive to the downright laughable.

For instance, the footage (above) of a deep-fake Tom Cruise makes you think twice about whether or not it’s actually real; while anyone who has seen a poor deep-fake application will know how eerie and unsatisfactory it can be to enter the Uncanny Valley.

It’s clear that in 2021, the quality of a deep fake depends on the amount of time someone can spend cleaning it up. That won’t always be the case, but it does undermine the speed at which deep fakes (at least great deep fakes) can be created.

Are digital humans synthetic media?

Digital humans are created using state-of-the-art CGI designed by teams of expert animators and visual effects specialists. The facial expressions are programmed and then autonomously animated by the digital human platform.

The conversations the vast majority of digital humans have today are also determined by conversational design experts using programs like IBM Watson, Dialogflow and Lex (the same tools that create chatbot dialogues).

So, by definition, digital humans as a technology don’t fit neatly into the bucket of synthetic media. They’re a type of conversational AI, rather than media created by AI.

However, parts of the digital human tech stack can absolutely include types of synthetic media.

For example, synthetic voice generation produced by Amazon Polly, WaveNet, Aflorithmic or WellSaid Labs can be considered synthetic media. As can language models like GPT-3 – tools and software that generate media without human involvement.

These technologies can provide content for your UneeQ digital human experience. For example, your digital human can use synthetic voice generation to say virtually anything you would want them to say, without having to record a human saying that same sentence. Tools like GPT-3 can make a digital human hold open-ended conversation without having to write each line of dialogue “by hand”.

We recently used such AI-driven tools to build both Digital Einstein and our GPT-3-powered digital human, Sophie.

Digital Einstein uses Aflorithmic’s tech to first clone the voice of Albert Einstein and then make it possible for him to say virtually anything, which is particularly helpful when he’s delivering dynamic content like his daily quiz.

Now, if you’re a brand looking at using synthetic media in business, it’s important to know that there are so many types and potential use cases, it’s hard to answer whether or not it is the best way to innovate. However, some broad pros and cons of synthetic media apply:

The pros of synthetic media

• Most content can be created extremely quickly with minimal human involvement.
• When it comes to using synthetic media in a consumer-facing way, they’re accessible 24/7, and the content can often be dynamic.
• The output is broad: synthetic media can incorporate writing, music, drawings, paintings, voice or visuals.
• The number of applications is also broad: synthetic media can be applied to apps, websites, gaming environments, VR/AR experiences and many more digital channels.
• They can be created fairly simply as either user-generated content or through third-party providers.

The cons of synthetic media

• There is less control over what is created, and said and done to the user. The AI is largely in charge of the quality and appropriateness of the output, making some forms of synthetic media risky for brands to incorporate.
• Deep fake technologies have difficult issues to overcome regarding trust. The general public is conscious of how it may be used to spread fake news or lead people to believe it is real when it’s not.
• Synthetic generation of voices and likenesses have been scrutinized for posing security issues, particularly around how they can bypass personal biometric security tools like facial or voice recognition software.
• Deep fake likenesses can often fall into the uncanny valley (meaning they look real but give off a feeling that something is wrong) leading users to disengage with the experience.
• There is an arguable lack of art and craft when it comes to AI-generated creative media, like music or paintings.

How do businesses use synthetic media?

Brands are using synthetic media today in a number of ways. However, due to the current risks of handing creative control over to AI (which cannot be ethical by nature) many of the safest consumer-facing applications still involve some form of human oversight.

For instance, our Digital Einstein experience was done in partnership with the Hebrew University of Jerusalem, who manage the rights of the Nobel Prize winner on behalf of his estate.

When it came to Sophie, our digital human who derives her conversational abilities from GPT-3, we established certain guardrails regarding what she could and wouldn’t talk about, with the help of the team at OpenAI. You can learn more about how we did so in this article, should it help you mitigate your own synthetic media challenges.

Similarly, the commercial use of deep fakes specifically have been focused on advertising and film, where the final output can be tightly controlled.

Brands like ESPN have brought legendary NFL rivals Al Davis and Pete Rozelle back to audiences using deep fake technology that allowed the faces of the deceased pair to animate without using traditional CGI.

Meanwhile, Hulu overcame social isolation restrictions to create an advertisement starring NBA player Damian Lillard, Canadian hockey player Sidney Crosby and WNBA player Skylar Diggins-Smith – all as deep fakes.

With everyone from the influencer industry to traditional advertising, film and TV getting involved, synthetic media will surely continue its push into the mainstream.

But as we like to say, customer experience is everything. How these synthetic media technologies are used will determine whether or not they provide the best experience to users.

Some will be curiosity projects, like AI-derived artworks; others will become part of the digital workforce, where the advantages of synthetic media can be integrated into more controllable commercial environments.

If you’d like to see what’s possible today with synthesised voice and conversation, you can speak to our digital human, Sophie, below. We’d love to hear how you get on!