What went into creating Vincent, a hyper-real digital human

Until very recently, directly mapping an actor’s performance onto photorealistic digital humans using real-time rendering was considered impossible. We’d seen the likes of Avatar and Alita in the movies. Still, these involved time-consuming offline rendering and post-processing work.
But thanks to the advancements in graphics hardware and software, as well as the relentless hard work from innovative teams in the field, we’re seeing real-time rendered digital humans. Meet Vincent, a digital human born from Korea-based creative studio, Giantstep.
Creating Vincent was not an easy feat. After much deliberation and research into potential joint partnerships, the studio’s R&D arm, GXLab took on the challenge to develop the technology in-house – with just five artists. To make Vincent a reality, the team quickly identified three key challenges they needed to overcome.

Skin and hair: the visual technology challenge
The first technical issue the team faced lay in ensuring that they had access to the necessary shading technology. Developing skin and hair features can cost an exorbitant amount of time, manpower and cost. With that in mind, the team turned to Unreal Engine. “Unreal Engine’s material editor and its powerful skin shading features like Transmission and Dual Lobe Specularity played a major role in boosting Vincent’s skin quality up to offline rendering levels without any additional development,” said Sungku Kang, Giantstep’s director of research and development.
The team made good use of Unreal Engine’s online learning courses to get to grips with the technology’s development processes and properties. “By leveraging all the available information, we were able to accurately understand how changing different parameters would affect the outcome. And use that information with more precise intent, rather than entering random numbers and leaving the results to chance,” Kang said. “Also, using material instancing to make immediate parameter changes and preview the results was very useful. This dramatically reduced the time spent in the final fine-tuning stage.”
The Meet Mike free asset sample provided by Epic Games also gave the team hints about the technologies they’d require to create the visuals for Vincent. Exploring the asset, they gathered necessary information, including how to express fine facial hairs and what kind of data is required for the final hair shape. “With all this information, our developers were able to reduce development time by clearly setting objectives for Maya plugins or scripts,” Kang explained.

Creating lifelike facial expressions in real-time
Choosing and implementing technologies for effective facial expressions was another challenge the team had to overcome. They found that most solutions were developed for offline rendering by default and were of low, video-game quality – which limited the level of detail or the ability to customise. They started researching the technology that was the nearest fit, and then based on their findings, they developed their own solution in-house.
Their first task was to assess which candidate offered an accurate three-dimensional location data while providing a high degree of freedom for the data format. They settled on Vicon Cara, a head-mounted camera apparatus, as the facial animation capture system that best met their needs. Using this device, marker locations can be set with great flexibility, and they can then be translated into three-dimensional data with extremely high accuracy.
“Most solutions at that time only scanned two-dimensional location information data of facial landmarks. The fact that Cara was able to capture three-dimensional data made it a good choice,” says Kang. However, Cara is for offline rendering, making real-time data transfer impossible out of the box. To resolve this, we decided to create a neural network that uses deep learning to infer 3D marker locations from 2D images.”
First, the team added one additional camera to Cara. When capturing the facial movements of the actor, video from the other camera was separately saved and used as the learning data. From this, the developers created artificial intelligence that can infer the 3D marker locations with high accuracy from a 2D image input. Machine learning was further leveraged in as many areas as possible, including for emphasising facial expressions and setting up blend shape weight. “Giantstep gained lots of experience in machine learning through this process. We were encouraged by the fact that the effective use of machine learning can help a small team overcome its limitations,” Kang said.

Optimising the digital human pipeline
The last major technical hurdle the team faced was in securing technology for optimising the pipeline. Given the small size of the team, they knew right away that efficiency was crucial. The most important initiative was to reduce the burden of manual iteration and to facilitate automation as much as possible. “This is another major reason why Unreal Engine was the best choice,” says Kang. “Unreal Engine’s support for Python and the convenience that brings to creating plugins allowed us to easily resolve iteration issues as well as to easily develop relevant tools.”
A case in point is the work the team did on photorealistic facial expressions. This process involved trying many combinations for the shape and number of areas into which the face is divided. And then followed by a preview of the results as quickly as possible. Because changing the facial areas required textures to be newly combined, assets had to be re-imported, and material composition details had to be readjusted. “If the whole process were to be manually handled, an artist would have to spend a full day on simple iterations to preview the results,” says Kang.
Instead, using Python scripting in Unreal Engine, the team was able to automate the management process of laborious tasks such as importing assets. “Tasks which previously required a full day whenever the data changed were automatically completed in just a few minutes,” continues Kang. “By maximising automation through various Maya plugins and scripts, and developing Unreal Engine plugins, Project Vincent was completed in a short time frame, despite the small size of the team.”

Beyond movies: the future of real-time digital humans
Real-time digital human development technology is expected to change the production of entertainment media that traditionally relied on offline rendering technology. It’s even predicted to move into other industries.
Synced with AI speakers or assistants, the technology could enable users to experience more intuitive AI services. We could start to see high-quality, hyper-real character creation used for effective cutting-edge marketing and promotional activities.
Giantstep intends to be a key part of this story, committed to advancing their technology and with plans to reveal its next development results as soon as possible.



Related posts

1Visionary Behind the Bionic Ear Honoured in 2024 Pause Awards Hall of Fame
3 December, 2024

Australia’s boldest innovators announced and new chapter for 2025
3 December, 2024

Top five winners score the most invaluable encounter
19 November, 2024

How Ally Watson is pioneering tech careers for women
15 November, 2024

Breakthrough Insights, Strategies, Creativity and Culture for now
12 November, 2024

Life lessons on ownership and how to keep creative control
6 November, 2024

How to unlock the intangible in brand equity with 4P’s
29 October, 2024

Discover trailblazing Finalists of Pause Awards 2024 in Australia
28 October, 2024

Why Bunnings feels like home: trust, community and genuine care
23 October, 2024

Why creativity and imagination will save the world
22 October, 2024

How Canva transformed a simple idea to become global leader
15 October, 2024

How NASA’s new AR tech will take astronauts to the next frontier
11 October, 2024

How to build authentic workplace culture
7 October, 2024

How to balance business growth with personal wealth
24 September, 2024

A time to dream big with Kristina Karlsson
16 September, 2024

The purpose of gatekeepers in authentic brand storytelling
6 September, 2024

It’s here, a final call to enter the Pause Awards this year
3 September, 2024

Join a free 7-week online learning on the go
3 September, 2024

Pause Awards partners with Ticker for exclusive broadcast coverage
6 August, 2024

Calling Melbourne — a significant player to enter!
6 August, 2024

Calling Hobart — an emerging hub to enter!
26 July, 2024

Calling Canberra — with most daring new ideas to enter!
16 July, 2024

Meet final judges and why they love Pause Awards
15 July, 2024

Meet even more judges and why they love Pause Awards
8 July, 2024

Meet more judges and why they love Pause Awards
5 July, 2024

Calling Adelaide — dynamic centre for innovation to enter
4 July, 2024

Meet the judges and why they love Pause Awards
1 July, 2024

Maybe those shouldn’t have been there in the first place
27 June, 2024

Calling Perth — an emerging innovator to enter!
25 June, 2024

Read a thrilling mid year predictions by the judges
24 June, 2024

Jasmine Batra’s Breakthrough Moment
24 June, 2024

WithYouWithMe’s Breakthrough Moment
24 June, 2024

Paz Pisarski’s Breakthrough Moment
20 June, 2024

UpStock’s Breakthrough Moment
20 June, 2024

Calling Brisbane — a rising star to enter!
19 June, 2024

How to choose the right category for the Pause Awards?
17 June, 2024

Success stories of past Pause winners
17 June, 2024

Your guide to the Pause Awards entry process
17 June, 2024

Understanding the breakthrough question
17 June, 2024

Top 10 tips on how to enter Pause Awards
17 June, 2024

Calling Sydney — a powerhouse of opportunity to enter!
6 June, 2024

Inke’s Breakthrough Moment
6 June, 2024

ReSource’s Breakthrough Moment
3 June, 2024

Brittany Garbutt’s exchange with Paul Bassat on building and sustaining a business empire
21 May, 2024

Entering Pause Awards for the first time?
21 May, 2024

Inside the Pause Awards 2024: A universe of possibility
14 May, 2024

What’s a Pause Awards breakthrough?
14 May, 2024

Marcella Romero’s Breakthrough Moment
9 May, 2024

Music Health’s Breakthrough Moment
2 May, 2024

Chau Le’s Breakthrough Moment
30 April, 2024

Tanck’s Breakthrough Moment
26 April, 2024

Macro Mike’s Breakthrough Moment
23 April, 2024

Birchal’s Breakthrough Moment
18 April, 2024

Tixel’s Breakthrough Moment
16 April, 2024

Redefining success with Angus and Neil of Tanck
15 April, 2024

Quad Lock’s Breakthrough Moment
12 April, 2024

Origin Energy’s Breakthrough Moment
10 April, 2024

Mel Stubbing’s breakthrough moment
5 April, 2024

Brittany Garbutt’s breakthrough moment
2 April, 2024

InvestorHub’s Breakthrough Moment
28 March, 2024

Kollektive’s Breakthrough Moment
26 March, 2024

Talking robots and AI Agents, two insane demos
25 March, 2024

Fight Club and Zombie VCs
25 March, 2024

WeMoney’s breakthrough moment
20 March, 2024

Addressing gender inequality with 21.7% discount for female-led companies
8 March, 2024

How Entertainment Brands are winning hearts and minds
5 March, 2024

2023 packed a punch
17 January, 2024

The power of relationships with Dom Pym at Pause Awards Night
19 December, 2023

Six years of celebrating the Australian most ambitious innovators
5 December, 2023

M&A green shoots & scandal central
10 November, 2023

Announcing the 63 ambitious finalists, 3 winners and Pause Awards Night
30 October, 2023

Can optimism and curiosity win the battle with Ai
20 October, 2023

Looking at 100 years from now in Solar, EVs and MedTech
20 October, 2023

What does the Australia’s VC landscape look like in 2050
19 October, 2023

Do we want to live in a Black Mirror world?
17 October, 2023

Four new emerging directors envision sustainable future beyond 2050
6 October, 2023

Visible Founders puts a spotlight on migrant entrepreneurs
28 September, 2023

Always be closing & IPO, no?
24 September, 2023

Strong commitment needed to run tech events in Australia revealed at GEC23
24 September, 2023

iPhone 15 event: everything about Apple’s new product line
14 September, 2023

Discover the colourful world of oral hygiene with Dsmile’s new range
13 September, 2023

How to build a company culture in dynamic market
8 September, 2023

The valuable startup lessons hidden in the film Oppenheimer
7 September, 2023

Meet the new addition to the Judging Board ‘23
4 September, 2023

Advice on Design Thinking for stellar product development
1 September, 2023

How Pause Awards can put your brand on the map
30 August, 2023

Seize the moment: New Extended Entry Deadline
30 August, 2023

I got 99 problems
26 August, 2023

Is AI the new frontier of creativity and business
25 August, 2023

Tech legends unite to guide growth and foster innovation
22 August, 2023

The new faces of Pause Awards in business and the product officers
21 August, 2023

Don’t miss out on the last chance to join champions
21 August, 2023

Experience is a new frontier for brands
18 August, 2023

How to grow an idea into a great product
11 August, 2023

Good storytelling and trust will fuel startup and brand growth
4 August, 2023

Twitter to X: a rebrand to challenge tech giants and empower users
4 August, 2023

Discover the latest tech predictions of 2023
1 August, 2023

Elevate your business with Stephen Hunt’s success secrets
31 July, 2023

Lean into digital marketing trends now and in 2024
28 July, 2023

The new faces of Pause Awards from agency and media innovation
25 July, 2023

Hit it, DJ! and other local newsings
24 July, 2023

Female-led ethical fintech Verve raises $3M for Verve Money
21 July, 2023

How to better tune into your success journey
21 July, 2023

Pause Fest’s BREAK–THROUGH SESSIONS will keep you scaling
12 July, 2023

How to choose the right awards for your business
5 July, 2023

The new faces of Pause Awards in strategy and leadership
4 July, 2023

We’re building the community for the most ambitious people
27 June, 2023

It was worth waiting for these unique features
26 June, 2023

Mapping the future of fit and function
23 June, 2023

Midnight Health secures $24 million funding boost
21 June, 2023

Championing breakthroughs in today’s business world
21 June, 2023

OpenAI’s Sam Altman says hai
19 June, 2023

Top 5 compelling reasons why you should enter
18 June, 2023

Pause Awards vs Webby’s, Cannes Lions and Effies
18 June, 2023

The new faces of Pause Awards in startup, product and experience
15 June, 2023

A tour of 11 new categories
12 June, 2023

Inside out of Pause Awards
5 June, 2023

Aim for the Diamond: Understanding the Different Categories
1 June, 2023

The new faces of Pause Awards in strategy, growth and innovation
30 May, 2023

Where business recognition gets a paradigm shift
24 May, 2023

Industry leaders predict a transformative year ahead
17 May, 2023

Pause Awards grows to $412 Billion ecosystem
17 May, 2023

Shaking it up with new Investable Score™
17 May, 2023

Tap into your inner powerhouse
5 May, 2023

Life in the fast (tech) lane
24 April, 2023

evee rides the electric wave
18 April, 2023

Big win for the female-led business Circle In, lands $2 Million
14 April, 2023

Designing global empires:
How local brands can conquer the world
10 April, 2023

A passionate pursuit for Healthcare reinvention
4 April, 2023

The game-changing sessions at SXSW Sydney
30 March, 2023

Top AI tools you need to try now
29 March, 2023

A journey to revolutionising healthy school lunches
28 March, 2023

Panic! at the SVB & metrics that matter
16 March, 2023

Call for Entries
opens on 17 May ‘23
1 March, 2023

Startup funding in decline, but early-stage hits peak in Aus
10 February, 2023

Music for wellbeing:
The healing power of sound
20 January, 2023

Deep dive with ChatGPT about Aussie tech ecosystem
17 January, 2023

How it started; How it’s going 2022
17 December, 2022

Learning from mistakes and how to tackle the upcoming headwind
14 December, 2022

Claiming your PauseNFT trophy
13 December, 2022

Women and purpose led businesses take the most wins at Pause Awards ‘22
6 December, 2022

Professor Fiona Wood – The pioneer of ‘Spray-on-Skin’ technique, ReCell.
6 December, 2022

Rise and Demise
28 November, 2022

Let’s have breakfast with champions?
15 November, 2022

82 Bold Finalists Announced for the Pause Awards ‘22
10 November, 2022

Judge Sessions with Lumigo, Tribal DDB and Clipboard Hospitality
24 October, 2022

Judge Sessions with LongView, Fullstack and Art Processors
20 October, 2022

The first look at the new wearable trophy design
13 October, 2022

The Public Voting is open, go get them!
10 October, 2022

Cash to splash & other VC news
2 October, 2022

The Final Deadline Extended to midnight 14 October
30 September, 2022

Judge Sessions with Wavia, SEIKK and MedTech Actuator
26 September, 2022

Pause Awards Wins Australia’s International Good Design Awards for Design Excellence
19 September, 2022

Judge Sessions with Synergy Group, AOK Creative and i4 Connect
16 September, 2022

A word with Birchal’s Co-Founder – Matt Vitale
6 September, 2022

What can Pause Awards bring to your company
30 August, 2022

Google, Synergy Group, Spaces Interactive, Safari and By Jacs judges for Culture categories
24 August, 2022

Loyal VC, Media.Monks, AOK Creative, Storyfolk and Irene Lemon judges for Good categories
15 August, 2022

Fullstack, Lumigo, Netambition, MedTech Actuator and Unhedged judges for Operators categories
10 August, 2022

A word with Heaps Normal’s Head of Brand – Peter Brennan
9 August, 2022

Tribal DDB, We Are Unity, Bullfrog, BeautifulAgile and The Audacious Agency judges for your Excellence
2 August, 2022

Simply Wall Street, IBM, Forestlyn, CFOWorx and Accenture Song are this year Growth Judges
1 August, 2022

Venture snapshot: down but not out
26 July, 2022

Best equity crowdfunding year in Australia – $86m!
22 July, 2022

How to navigate the awards Entry Portal?
14 July, 2022

Startup Genome ecosystem report 2022 review
10 July, 2022

We’re giving a voice to our community with a stylish newsroom design
10 July, 2022

Professor Martin Green
– The father of solar cells
10 July, 2022

Dr John O’Sullivan
– The inventor of modern WiFi
10 July, 2022

14 New award categories under five tracks to highlight the ecosystem success
5 July, 2022

M&A on the rise & the EV SPAC demise
29 June, 2022

Our new vision:
The home for champions
27 June, 2022

Awards LIVE Briefing and Entry Kit
22 June, 2022

The Porsche of awards programmes with community at its heart
21 June, 2022

The Sheet Society
– Success Stories
17 June, 2022

We’re looking for our final few judges – could you be one?
8 June, 2022

Single Use Ain’t Sexy
– Success Stories
1 June, 2022

Afterpay
– Success Stories
24 May, 2022

This is how we celebrate innovation champions
15 December, 2021

The most ambitious and forward-thinking companies in Australia revealed
24 November, 2021

Pause Awards 2021 Finalists Skyrocket
17 November, 2021

Final Deadline is Looming
1 September, 2021

Get to know:
Carolyn Breeze – Judge
24 August, 2021

Get to know:
Will Hayward – Judge
17 August, 2021

Get to know:
Shamila Gopalan – Judge
10 August, 2021

Get to know:
Tom Leyden – Judge
3 August, 2021

Get to know:
Jules Brooke – Judge
26 July, 2021

Get to know:
Jamie Finnegan – Judge
18 July, 2021

Early Entries end This Week
12 July, 2021

Need Help Entering Pause Awards?
6 July, 2021

2021 Award Category Guide
1 July, 2021

How to Enter The Pause Awards in 7 min?
25 June, 2021

Get to know:
Melanie Rayment – Judge
24 June, 2021

Past winners:
where are they now?
24 June, 2021

Pause Awards 2021 opens for entries
9 June, 2021

New categories revealed for Pause Awards 2021
5 June, 2021

Meet the Judging Board 2021
13 May, 2021

The Reign in the North
1 February, 2021

Dane O’Shanassy on Patagonia’s moral compass and commercial success
6 November, 2019
Pause index

Slyp
I Wish I'd Done That

Removify
Defiant Ones, I Wish I'd Done That

Law On Earth
B-Good

Ovira
Out Of The Garage, I Wish I'd Done That, Hammer

Vanessa Doake
Women In Business

Upstock
Design First, Down Under

Music Health
B-Good, Hammer, On The Rise, I Wish I'd Done That

LunchFox
I Wish I'd Done That

Seedling Giving
B-Good

Mr Yum
Hammer, Work Sweet Work, Defiant Ones

Melinda Farley
Women In Business
