Performance Capture and the Virtual Environment: Supplanting Physicality
The advent of digital has occasioned many changes in the way film is made and viewed; many directors now film digitally rather than on film stock, and thus editing is now a primarily computerised process. Lines are blurred as television, characterised by its role as a private, home-based viewing medium with episodic content, traditionally shot on a lower budget and with lesser equipment than cinema, grows and begins to match the silver screen in both technology and content. Services such as Netflix and LoveFilm which offer us the option of streaming content from both film and television on computers, mobile phones, and tablets subvert the communal cinema experience as viewing becomes ever more individual, ever more private. The emphasis on how we receive content has shifted, too, as these services provide programming on demand, in some cases releasing a whole series simultaneously to be viewed at the subscribers discretion.
As the technology has evolved, the filming process has changed as much as the viewing process; filming on digital, takes are no longer governed by the ten minute limit of a reel of film. The day’s film no longer needs to be developed over night in order for footage to be reviewed, but can be played back instantly. Similarly, the cinematographer can see how his shots turn out right there and then and decide if he needs to re-shoot, rather than waiting, wondering and being forced to try and recapture the moment the next day. As many directors abandon celluloid in favour of digital, film becomes less alchemical and more technological.
Special effects, which used to be the realm of the likes of Ray Harryhausen and his golemic stop-motion puppets, are now dominated by animators working with computers and render farms. Less and less of what we see on screen is real in the physical sense – given the fictional nature of film, it would be inaccurate to describe anything on screen simply as ‘real’ – as more and more is left to digital insertion post-filming. Where once directors would have made use of puppets, miniature sets, prosthetics and pyrotechnics, now they make use of the numerous powerful tools afforded them by digital technology.
Directors such as James Cameron and George Lucas have taken a direct hand in creating the technology they deem necessary to tell their stories; dissatisfied with the tools available to him at the time, Lucas began his Star Wars saga halfway through, later hiring Ed Catmull to head Lucasfilm’s Computer Graphics Division. This component of Lucasfilm would later go on to be purchased by Steve Jobs and become Pixar, the company which produced the world’s first fully computer animated feature length film, Toy Story, in 1995.
Toy Story was nominated for several awards, and won its creators awards for leadership and technological achievement. It also proved that computer animation was a viable approach to filmmaking and paved the way for a slew of CG features and effects. A mere six years later, Hironobu Sakaguchi, creator of Final Fantasy, brought Final Fantasy: The Spirits Within to our screens in 2001. Although a box office bomb which returned only $85 million of its $137 million budget, it was notable for being the first photorealistic computer animated feature film. Bearing little resemblance to the franchise from which its name is derived, Spirits Within was crippled by its rather tepid plot and somewhat bland characters; although Aki and her comrades look fantastic, they occupy a place on the slopes of the Uncanny Valley.
Although, or perhaps because, the film’s characters look almost real, they maintain something of an alien quality, whereas the endearing cast of Toy Story possessed just enough human characteristics to give a familiar sense of humanity, of life, without striving to seem truly alive, audiences were able to become emotionally invested in them. Because the characters of Spirits Within strive for true photorealism, we are more acutely aware that there is something not quite human about them.
Sakaguchi’s film made extensive use of motion capture technology in its attempt to make its characters as realistic as possible, actors being filmed performing scenes from the film and their movements used as rigs to animate their digital counterparts. This technique allows animators to give their models movements which appear as life like as possible, an important step in achieving realism in a CG film. The movements are indeed lifelike, and there are moments of the film where it is possible to forget that what we are seeing is not live action, but despite the care taken in animating each individual hair on Aki’s head, the fact remains that the closer to human these homunculi get, the more we become aware that they are not.
James Cameron, when making his long conceived sci-fi epic Avatar, was keenly aware of this problem; relying as it does on the concept of the avatar, lab-grown specimens of Pandora’s indigenous Na’vi that the human characters can inhabit, it was necessary that these digital counterparts recognisably resemble their actors. Indeed, it is important to the themes and success of the film that the Na’vi as a race be human enough that audiences would sympathise with them against the invading human characters. Unfortunately, initial attempts at rendering Sam Worthington as an avatar fell drastically short of the humanity Cameron intended.
To make the whole thing work, Worthington’s performance, those subtle expressions that sell a character to the audience, had to come through the face of his Avatar. But after millions of dollars of research and development, the Avatar‘s face was not only lifeless, it was downright creepy. It “scared the crap out of me,” Cameron recalls. “Horrible! It was dead, it was awful, it wasn’t Sam. God, I thought. We’ve done everything right and this is what it looks like?”
Of course, true to his reputation as a technical innovator, Cameron overcame this issue, mounting small cameras on headgear for his actors to wear so that every nuance of their facial expressions could be captured in close up and used to map the faces of their avatars. In the same way that the ‘mocap’ suits are covered in markers for reference, the actor’s faces were painted with an array of dots so as to create a comprehensive rig which could be used to animate their digital incarnations.
This method was much more successful, as the finished film attests, but it was not the only innovation Cameron made during the filming of Avatar. Aiming to immerse his audience as fully as possible in the film, Cameron used a modified version of the Fusion Camera System to film in 3D, giving the world of Pandora a depth which draws its audience in. Dissatisfied with the practice of choosing shots and angles on a computer screen in postproduction, Cameron charged his virtual-production supervisor to provide him with a way of filming his actors in the virtual environment of Pandora.
The result was the ‘swing camera’, a lensless device which Cameron could walk around the motion capture set, called the ‘volume’, and see a low resolution image of how the finished shot would look. The camera works with the volume, feeding information about its position in relation to the actors through a computer to return a rough version of the finished CGI scene to its monitor. The actors are free simply to act without worrying about the camera, and Cameron is able to reshoot scenes from different angles simply by walking around the empty volume; the scene is already recorded, so the actors don’t need to recapture the moment and the immediacy of the live shooting is not lost.
Only a third of Avatar at most was filmed to be completely live action, the rest consisting of virtual environments. The technology may be expensive, but its merits are evident; worlds can be created on a scale and level of detail never before possible, actors can be transformed beyond the means of the means of prosthetics with relative ease, and scenes can be shot without the actors being present. With true photorealism almost within reach, the limitations on computer animation and what it can do for sci-fi and fantasy films are few.
Just as filming in digital has removed the limitations of ten minute film reels, motion capture allows actors the freedom to simply act within a space without worrying where the camera is or will be, without having to spend hours in hair and make-up. Stunts and environments that would be extremely difficult, if not impossible to create practically in reality, computer animation can achieve with relative ease.
Although Spirits Within was not entirely successful in its attempt to be the first feature length computer animated film with photorealistic human characters, it came extremely close and was lauded for its technical achievement if not its storytelling. The spectacle of Avatar shows how far the technology has come and what it can do, and whilst creating a completely photorealistic human character may still present difficulties, the day where it becomes possible to do so cannot be far off.
The question then arises of the role of the physical in filmmaking, of live action; if creating photorealistic human characters becomes a process of relative ease, as the creating of virtual environments already is, the need for live action may potentially diminish. Indeed, Square intended Aki Ross to become the first animated actress, to give her roles in multiple films in different personas. If motion capture technology continues to evolve as it has, it is conceivable that rigs, once created, might be reused outwith their original projects, giving ‘life’ to character models as digital actors.
Locations need not be scouted, merely sculpted; the perfect scene need be captured only once and the camera angles and movements chosen or altered later with the likes of Cameron’s ‘swing camera’. Digital actors would still need to be voiced, of course, but it is not inconceivable that a voice library might be built for a character just as its movements are given to it. All of this, of course, is speculative projection; for the time being, CG films are still largely rooted in the physical, if only for reference in some cases.
Cameron chose the cast of Avatar as much for their physical appearances as their vocal talents, and they were trained in archery and horse riding in order to make their performances as Na’vi as realistic as possible. Despite the rainforest scenes of the film being created digitally, the cast were taken to Hawaii to get a feel for the sort of environment their characters would be inhabiting. This is why Cameron refers to the motion capture sequences of his film as ‘performance capture’ – it is not just the characters movements he captured, but their facial expressions, every nuance of their immersion in the roles they were cast.
For the time being computer animation is used largely as an augmentation, or a stylistic choice, but given the rate that such technology advances and the transition of other entertainment mediums into digital, it stands to reason that its presence will only grow with time. By no means do I intend to suggest that live action and physicality will be replaced altogether, simply that digital is on the rise, and perhaps one day the animated may be indistinguishable from the not.
What do you think? Leave a comment.
Important topic, great read! Was expecting you to mention S1m0ne somewhere. There were a lot of controversies when this was produced. For example (from IMDB trivia):
Then how did the execs react to Virtuosity from 1995? 🙂
Even if all the expressive components involved in human acting—physical presence, voice, gesture, posture, facial expression, the unique qualities with which an individual moves, etc.—could be replicated individually and without reference to actual human beings, could these individual components be assembled into a whole that would be perceptually the same as what you get from an actual actor? I sort of doubt it. So what you would get with post-human cinema would be something entirely different, a sort of geometrical abstraction of humanity, which might be interesting… but, then again, maybe it wouldn’t be.
We have the technology, yes , but would it look realistic enough to fool people into believing that it is real? Yeah the Navi looked pretty good, but the are “creatures” nonetheless. Animals in CGI and such look very close to realistic, but we can always tell it isn’t, by how it moves, acts, reacts, sits down, stands up, etc.
Now try to CGI a human being, AS IS. No 5 second scene with some sort of impossible stunt, no blurrs, no glares, no NONE of that.
We are getting there, BUT we are not there yet.
Very current discussion indeed. I would add that one shouldn’t simply think of this in terms of what would be lost, or what we are familiar or comfortable with now, but what can be gained by this, and that is a sort of elasticity which comes when one can so readily shift what is “real”. The use of CGI is already growing enormously when it comes to shaping all the other parts of the screen for this reason, so I’m not sure the actors can be held as wholly exempt from this. Bringing up Cameron, for example, should also bring to mind how hugely successful his films are, so there is some audience for this sort of shift, the question will be how wide spread will the demand or acceptance become and what the mass audience feels about the possible trade-offs it may entail. If someone doesn’t notice the individual expressions or unique presence of the actors in the first place, then the substution will be much less jarring than to anyone who values it.
I think in 50 years, animation will coexist with live action, much the same way that it did in the past and how it is currently in the present.
CGI is awesome, and once in a while I love being wowed by a VFX extravaganza, but the real world is much more fun to film in. If the end result would look identical, and money weren’t an issue either way, I prefer in-camera effects simply because they’re more fun to do. If I were acting, I’d rather rappel off a real building than let a stunt double in a mocap suit do it on a green screen in some studio. If I were directing, I’d rather be able to see the actors interact with the world in real time. It’s just more fun that way.
As an audience, I don’t care whether something is done for real or on a computer. The problem is, CGI is often overdone, and that’s just as annoying as an actor who’s overdoing their performance. I wish effects were more subtle.
I agree, the real world can offer more appealing and simpler (not to mention cheaper) solutions to CGI.
Really fascinating article; I learnt a lot. It’s especially interesting to hear about Avatar.
This is just a next step in what film has been doing since before sound. The first big leap though was rotoscoping, on display very effectively in animation by the Fleischer Studio in the 1930s. Most of the human characters were rotoscoped, which really was a very primitive form of MoCap.
Capture on film rather than as a digital file, the covering or replacing the original performance with something artificial.
As long as the modern tech does not become the crutch that overwhelms the film, like when movies like Sky Captain and the World of Tomorrow become all about the tech used rather than the story, this new tech used for an old movie trick can only allow filmmakers to more accurately reproduce their vision.
Like every other technology, it will be abused and misused once it becomes more cheaply available, and that is when we as consumers, will need to watch carefully and vote with our dollars.
Well, now I know what the Uncanny Valley is!
While this transformation of animation and CG is revolutionary for the film industry, I have to wonder about what we’re losing in the meantime. Sure, prosthetics take hours to apply and may not look so great, but it’s something about the physical presence and reality of it that attracts me. If worlds can be made on a computer, there is no need for scouts to find a location. If characters can be generated pixel by pixel, live actors are going to have even more trouble finding work (even though they will still need voices). I believe that this advancement of technology is great, but to a point. Great article – it made me see animation in a new light!
I hardly ever watch movies that have a lot of fx, so when I saw Oblivion, I was blown away with the look of the non-real parts of the film. I could very well grow to appreciate spectacular vistas that spring from the minds of talented fx people.
Very informative read! It was interesting to learn about the making of Avatar and Cameron’s choice to immerse the actors in a rainforest-type environment to get a better sense of their characters’ reactions. Your point that “The actors are free simply to act without worrying about the camera” due to digitization made me think about the role of the actor as an actor — that is, the profession of acting. Isn’t considering the camera’s presence an essential part of screen-acting? Being aware of an audience is a huge element to extract through digitization. Will the acting get “better” because of this?
I liked the comparison to Toy Story. There is a definite sense of humanity to its characters. So much so that even as an adult it doesn’t really register that they are in fact animated characters. But I wonder how the strength of storytelling helped.
I have not seen Avatar but the complaints I’ve heard from it all have to do with the cliche storytelling. So as I understand it, animation needs to be top notch to sell the characters. But the storytelling has to be up to par as well.
Great article! I didn’t realize there was a Final Fantasy movie (or perhaps I knew at one point but just dismissed it the way I typically dismiss video game movies), and I looked up the graphics from it, and they’re actually pretty impressive for 2001. You’re right, though, in saying they have an uncanny valley effect.
Very good article! Esp. enjoyed the material on Avatar. You really gave insight into how hard Cameron and Co. had to work to make it look good.
I don’t think that Cameron’s animation efforts, titanic (lol) as they may have been, probably would not be repeated.