Performance Capture and the Virtual Environment: Supplanting Physicality

The advent of digital has occasioned many changes in the way film is made and viewed; many directors now film digitally rather than on film stock, and thus editing is now a primarily computerised process. Lines are blurred as television, characterised by its role as a private, home-based viewing medium with episodic content, traditionally shot on a lower budget and with lesser equipment than cinema, grows and begins to match the silver screen in both technology and content. Services such as Netflix and LoveFilm which offer us the option of streaming content from both film and television on computers, mobile phones, and tablets subvert the communal cinema experience as viewing becomes ever more individual, ever more private. The emphasis on how we receive content has shifted, too, as these services provide programming on demand, in some cases releasing a whole series simultaneously to be viewed at the subscribers discretion.

As the technology has evolved, the filming process has changed as much as the viewing process; filming on digital, takes are no longer governed by the ten minute limit of a reel of film. The day’s film no longer needs to be developed over night in order for footage to be reviewed, but can be played back instantly. Similarly, the cinematographer can see how his shots turn out right there and then and decide if he needs to re-shoot, rather than waiting, wondering and being forced to try and recapture the moment the next day. As many directors abandon celluloid in favour of digital, film becomes less alchemical and more technological.

Special effects, which used to be the realm of the likes of Ray Harryhausen and his golemic stop-motion puppets, are now dominated by animators working with computers and render farms. Less and less of what we see on screen is real in the physical sense – given the fictional nature of film, it would be inaccurate to describe anything on screen simply as ‘real’ – as more and more is left to digital insertion post-filming. Where once directors would have made use of puppets, miniature sets, prosthetics and pyrotechnics, now they make use of the numerous powerful tools afforded them by digital technology.

Directors such as James Cameron and George Lucas have taken a direct hand in creating the technology they deem necessary to tell their stories; dissatisfied with the tools available to him at the time, Lucas began his Star Wars saga halfway through, later hiring Ed Catmull to head Lucasfilm’s Computer Graphics Division. This component of Lucasfilm would later go on to be purchased by Steve Jobs and become Pixar, the company which produced the world’s first fully computer animated feature length film, Toy Story, in 1995.

Toy Story was nominated for several awards, and won its creators awards for leadership and technological achievement. It also proved that computer animation was a viable approach to filmmaking and paved the way for a slew of CG features and effects. A mere six years later, Hironobu Sakaguchi, creator of Final Fantasy, brought Final Fantasy: The Spirits Within to our screens in 2001. Although a box office bomb which returned only $85 million of its $137 million budget, it was notable for being the first photorealistic computer animated feature film. Bearing little resemblance to the franchise from which its name is derived, Spirits Within was crippled by its rather tepid plot and somewhat bland characters; although Aki and her comrades look fantastic, they occupy a place on the slopes of the Uncanny Valley.

Although, or perhaps because, the film’s characters look almost real, they maintain something of an alien quality, whereas the endearing cast of Toy Story possessed just enough human characteristics to give a familiar sense of humanity, of life, without striving to seem truly alive, audiences were able to become emotionally invested in them.  Because the characters of Spirits Within strive for true photorealism, we are more acutely aware that there is something not quite human about them.

Sakaguchi’s film made extensive use of motion capture technology in its attempt to make its characters as realistic as possible, actors being filmed performing scenes from the film and their movements used as rigs to animate their digital counterparts. This technique allows animators to give their models movements which appear as life like as possible, an important step in achieving realism in a CG film. The movements are indeed lifelike, and there are moments of the film where it is possible to forget that what we are seeing is not live action, but despite the care taken in animating each individual hair on Aki’s head, the fact remains that the closer to human these homunculi get, the more we become aware that they are not.

James Cameron, when making his long conceived sci-fi epic Avatar, was keenly aware of this problem; relying as it does on the concept of the avatar, lab-grown specimens of Pandora’s indigenous Na’vi that the human characters can inhabit, it was necessary that these digital counterparts recognisably resemble their actors. Indeed, it is important to the themes and success of the film that the Na’vi as a race be human enough that audiences would sympathise with them against the invading human characters. Unfortunately, initial attempts at rendering Sam Worthington as an avatar fell drastically short of the humanity Cameron intended.

To make the whole thing work, Worthington’s performance, those subtle expressions that sell a character to the audience, had to come through the face of his Avatar. But after millions of dollars of research and development, the Avatar‘s face was not only lifeless, it was downright creepy. It “scared the crap out of me,” Cameron recalls. “Horrible! It was dead, it was awful, it wasn’t Sam. God, I thought. We’ve done everything right and this is what it looks like?”

Of course, true to his reputation as a technical innovator, Cameron overcame this issue, mounting small cameras on headgear for his actors to wear so that every nuance of their facial expressions could be captured in close up and used to map the faces of their avatars. In the same way that the ‘mocap’ suits are covered in markers for reference, the actor’s faces were painted with an array of dots so as to create a comprehensive rig which could be used to animate their digital incarnations.

This method was much more successful, as the finished film attests, but it was not the only innovation Cameron made during the filming of Avatar. Aiming to immerse his audience as fully as possible in the film, Cameron used a modified version of the Fusion Camera System to film in 3D, giving the world of Pandora a depth which draws its audience in. Dissatisfied with the practice of choosing shots and angles on a computer screen in postproduction, Cameron charged his virtual-production supervisor to provide him with a way of filming his actors in the virtual environment of Pandora.

The result was the ‘swing camera’, a lensless device which Cameron could walk around the motion capture set, called the ‘volume’, and see a low resolution image of how the finished shot would look. The camera works with the volume, feeding information about its position in relation to the actors through a computer to return a rough version of the finished CGI scene to its monitor. The actors are free simply to act without worrying about the camera, and Cameron is able to reshoot scenes from different angles simply by walking around the empty volume; the scene is already recorded, so the actors don’t need to recapture the moment and the immediacy of the live shooting is not lost.

Only a third of Avatar at most was filmed to be completely live action, the rest consisting of virtual environments. The technology may be expensive, but its merits are evident; worlds can be created on a scale and level of detail never before possible, actors can be transformed beyond the means of the means of prosthetics with relative ease, and scenes can be shot without the actors being present. With true photorealism almost within reach, the limitations on computer animation and what it can do for sci-fi and fantasy films are few.

Just as filming in digital has removed the limitations of ten minute film reels, motion capture allows actors the freedom to simply act within a space without worrying where the camera is or will be, without having to spend hours in hair and make-up. Stunts and environments that would be extremely difficult, if not impossible to create practically in reality, computer animation can achieve with relative ease.

Although Spirits Within was not entirely successful in its attempt to be the first feature length computer animated film with photorealistic human characters, it came extremely close and was lauded for its technical achievement if not its storytelling. The spectacle of Avatar shows how far the technology has come and what it can do, and whilst creating a completely photorealistic human character may still present difficulties, the day where it becomes possible to do so cannot be far off.

The question then arises of the role of the physical in filmmaking, of live action; if creating photorealistic human characters becomes a process of relative ease, as the creating of virtual environments already is, the need for live action may potentially diminish. Indeed, Square intended Aki Ross to become the first animated actress, to give her roles in multiple films in different personas. If motion capture technology continues to evolve as it has, it is conceivable that rigs, once created, might be reused outwith their original projects, giving ‘life’ to character models as digital actors.

Locations need not be scouted, merely sculpted; the perfect scene need be captured only once and the camera angles and movements chosen or altered later with the likes of Cameron’s ‘swing camera’. Digital actors would still need to be voiced, of course, but it is not inconceivable that a voice library might be built for a character just as its movements are given to it. All of this, of course, is speculative projection; for the time being, CG films are still largely rooted in the physical, if only for reference in some cases.

Cameron chose the cast of Avatar as much for their physical appearances as their vocal talents, and they were trained in archery and horse riding in order to make their performances as Na’vi as realistic as possible. Despite the rainforest scenes of the film being created digitally, the cast were taken to Hawaii to get a feel for the sort of environment their characters would be inhabiting. This is why Cameron refers to the motion capture sequences of his film as ‘performance capture’ – it is not just the characters movements he captured, but their facial expressions, every nuance of their immersion in the roles they were cast.

For the time being computer animation is used largely as an augmentation, or a stylistic choice, but given the rate that such technology advances and the transition of other entertainment mediums into digital, it stands to reason that its presence will only grow with time. By no means do I intend to suggest that live action and physicality will be replaced altogether, simply that digital is on the rise, and perhaps one day the animated may be indistinguishable from the not.

What do you think? Leave a comment.