While you were celebrating the reunion of Robert De Niro, Joe Pesci, and Martin Scorsese, and the impossible bliss of one of our greatest living directors delivering his ultimate statement on his most beloved genre, you missed the real revolution. The Irishman has changed everything, and cinema will not be the same going forward. A new method of performance capture has arrived, and it’s a most artistically liberating technological breakthrough. Mocap apes, your rise, dawn, and war is over. Goodbye green dots and skin-tight scuba suits, hello Al Pacino in all his naked glory.
Visual Effects Supervisor Pablo Helman has been chasing the Holy Grail of his industry for more than two decades. How do you convince an audience member that a human is a human when it is not? We love to spot a fake. We cherish calling bullshit on an off-balanced Spider-Man swinging through New York City, or a Will Smith Gemini Man creeping out of that uncanny valley. Meeting that crossed-arm theatergoer in their realm, and not just blowing their mind, but utterly fooling it, is nearly impossible. For now.
“The whole thing about digital humans is performance,” Helman explains during our recent conversation. “If we don’t have the performance, then the likeness is never going to be there.” Your software may be top of the line, but it’s all for not if the program cannot be fed the genuine article.
Nailing the behavioral likeness of a person before The Irishman involved a swarm of sensors placed upon the performer, confining them to an absurd costume of technology, and requiring an extraordinary suspension of disbelief. You’ve all seen the Avengers: Endgame behind-the-scenes footage; Mark Ruffalo and the cast around him practically have to enter another reality to pull an authentic Hulk into being.
For The Irishman to work, de-aging was essential. However, there is a reason why you haven’t seen these actors galivanting around the MCU. “We were talking with Marty,” says Helman, “and one of the things he said was, ‘I’ve seen these markers and these ping pong balls. I don’t think Bob De Niro is going to go for it. Al Pacino and Pesci are definitely not going to go for it. They’re method actors. They don’t want any interference with their work.'”
These actors had to be left alone. The technology had to be removed from their process. “So with ILM [Industrial Light and Magic], and a bunch of people in their R&D department,” says Helman, “we asked, ‘If we don’t have markers, how can we grab the performance? What is it that is left?’ The answer to what’s left is lighting and texture.”
Helman and ILM, in collaboration with cinematographer Rodrigo Prieto and the Arri Group, created a new three-headed camera rig. The set-up contains one primary Red Helium camera alongside two witness cameras adapted for infrared cinematography. The images the cameras capture are filtered into ILM’s Flux software and it transforms them into 3D renders of the actors’ facial performances. The data is then placed over 3D models of each actor, which fabricates a digital replica of the performance recorded on set. The information is then manipulated to create a youthful likeness.
“It’s very much a science thing in which the software takes a look at the lighting and registers all the lighting instruments and the intensity and the color of the light,” explains Helman. “It then takes a look at the digital double version of the 76-year-old Bob De Niro and compares the neutral pose with what’s happening in the background, on the plate that we capture, and then deforms the geometry to match the background. Then it does the same thing frame by frame by frame until it creates a scene. It’s not something that we have done before, but it’s something that we tried during a test in 2015, and it worked. We just spent two years trying to make the production model of that technique so that we could make the movie.”
On top of all the technology required to reproduce the performances effortlessly, the camera unit had to be no wider than 32 inches so that it could pass through doors, and it could weigh no more than 64 pounds, which is the limit of the Technocrane. “If you take the burden away from the actors, you put the burden somewhere else,” says Helman. “The first department that gets burdened is the camera department because now you have three times the amount of cameras and you also have double the amount of people that you have in the camera department. Then on top of that, unprecedentedly, the visual effects team is embedded into the camera department.”
Helman spent four years on The Irishman, and 108 of those days were spent on set. The experience radically enhanced his perception regarding his work. “For me, it was an incredible discovery of understanding what the actors go through and being absolutely respectful of their art,” says Helman. “Ambiguity is one of the most human things that is very difficult to get. With your eyes, you’re saying one thing, but your mouth is saying another. That’s acting! The computer doesn’t understand ambiguity. In CG, the difference between a wince and a smile is just a couple of pixels.”
Replicating ambiguity is what kept the visual effects supervisor up at night. He spent an inordinate amount of time negotiating with the computer, fluctuating wrinkles, and educating the system on an infinite number of biological quirks. Such deliberation was made all the more difficult because Scorsese did not want to fall back on keyframe animation, requiring every issue to be solved by the computer.
“The reason why that was important was that Marty didn’t want to change any of the performances,” Helman says. “It was a real process for Marty to understand what makes somebody younger without taking the performance away and keeping the weight of the feeling of those performances alive.”
In addition, the 25-year-old Frank Sheeran is not simply a 25-year-old version of Robert De Niro. Helman and his team altered eye color to match the character as well as the width of De Niro’s face. Scorsese did not want to rewind the clock on the actor and discover Jimmy Conway from Goodfellas. Frank Sheeran is not Jimmy Conway, and he is not Robert De Niro. He’s a totally unique individual, and that’s who Scorsese wanted to see when The Irishman hopped around its timeline.
“I remember when I saw The Untouchables in 1987,” says Helman. “I saw Robert De Niro play Al Capone. You see him there, and that is not Robert De Niro. He looks heavier, and he doesn’t have hair, but it’s not like the audience is saying, ‘Wait a minute, I want to see Taxi Driver!’ He’s still the character.”
It’s the same deal with The Irishman. Helman, Scorsese, and De Niro are designing a person from the ground up. You find Frank Sheeran in his clothes, his mannerisms, and his facial features. In a way that he has never done before, Helman took on his work from the actor’s perspective, not necessarily considering the regular visual effects conversation.
What Helman and his team have created on The Irishman is a groundbreaking system with infinite potential. “I know it sounds big for me to say this is going to change the way we make movies,” he says, “but I think that taking the markers and the technology away from the actors helps the performances.”
De-aging is not the revelation. A spring-chicken Robert De Niro is merely the result of the way Helman captured performances on set. But that performance could be re-targeted onto any number of creature designs that have nothing to do with the human shape. Mark Ruffalo could finally be free of the humiliating barbs of his co-stars. The next generation of digital beings will be indistinguishable from those humans that stumble around your daily life. Thank you, Mr. Helman.