Future Magic (III. Creative Uses for AI)

(Note: this essay is part of a series.)

In 2007 a restored version of Blade Runner (1982) was released. Part of the restoration was changing the head of an obvious stuntwoman to that of the actress, Joanna Cassidy. The faces clearly didn’t match and neither did the wigs. The restoration crew got Cassidy into a greenscreen studio and filmed her performing the required head and face contortions. They could have simply plastered that footage on top of the original, but the stuntwoman’s wig would be visible underneath. So first, they digitally removed the stuntwoman’s entire head from the footage! You can glimpse the resulting intermediary version here.

When I saw this in 2007, I was stunned. It was one of the most amazing feats of technical wizardry I had ever seen. It seems as if they had software that could analyse footage and work out, if a particular element were removed, what would now be visible “behind” it given the surrounding context. This was amazing because the surrounding context in this case was extremely complex, with lights and objects shifting around in different directions, and most of them out of focus. I am still baffled as to how this was achieved, especially in a pre-AI era.

Eight years later I saw a similar thing with the remake of Robocop. The original was made in 1987 when it was impossible to do any of the sort of augmentation we will discuss, so the character looked like a man wearing a huge bulky suit rather than an actual cyborg. In 2014, it was possible to slim everything down after the fact, so that it looked like the cybernetics were his actual body, not something wrapped around it. The effect is incredible:

Notice that, as in the Blade Runner effect, the background - what would be visible if he were slimmer - is somehow being “restored”. Look especially on either side of his waist, where the floor that was hidden in the filmed shot is somehow “restored”. It’s possible this was achieved by filming the camera move through the set without the actor present, then repeating the exact same camera move with him there, then strategically combining the two shots. But it’s also possible that software was “guessing” the background.

Disregarding that question about the background, what is certain is that almost every element of his cybernetic suit is augmented - reshaped, tightened up, replaced. If you lay the two versions on top of each other and switch between them, you can see that almost no element of the original figure remains in the finished shot, which therefore is effectively CGI, but one guided very tightly by real footage. Clearly, the software had reached a point where it could discern shapes and then “work with them”.

And again, this was without AI as we know it today. We are now in a very different era. To give a flavour, here is a YouTube comment about Alien 3, directed by David Fincher in 1992, beleaguered by a studio who kept interfering and forced him to make a film he wasn’t proud of:

Believe it or not, Alien 3 may yet actually get a director’s cut at some point... I can’t get into details except to say that I know for a fact the idea is floating around in David Fincher’s direct circle... You’ve got to remember guys... it’s 2025: the limitations of what can and can’t be done to finish a decades old film simply don’t exist anymore if you’re willing to take the time and effort to do it.

What he means, apart from anything else, is that scenes which Fincher wanted but was prevented from filming in 1992 could now be created digitally, even “with” actors who have died in the interim. As he says, it wouldn’t be easy. It would take time and persistence - but it is now possible.

AI will enable augmentation and “resynthesis” of material - both visual and aural - in ways that have never been possible before.

Switching language

The era of clumsy dubbing could be ended by the ability to automatically change dialogue to a different language. This involves both resynthesising the actor’s voice to produce different words, and visually changing their mouth movements to match. It is already possible.

Changing dialogue

Dialogue can be changed after the fact, whether to remove swear words (to make the drama suitable for a wider audience) or to fix mistakes made in performance or to align with a later version of the script. Again, this is already possible.

Fixing errors

Think of the possibilities for repairing or slightly altering a shot taken in the traditional way. If this technology had been available to Kubrick, he wouldn’t have needed to do innumerable takes of every shot until he got exactly what he wanted. Slight errors or imperfections, whether by the actors or the crew, could have been fixed very easily with AI. Even errors that would have confounded the best image technicians of 2015 would present little trouble for a well-trained AI today. As a result, Kubrick would have made his films much faster. In fact, if this technology had been available to him, we might have twice as many Kubrick films as we do have. Does that mean they would be half as good? I see no reason to assume so. What always matters is the drive and intent of the creator. If Kubrick went about his craft in the way that lazy AI creators go about theirs today, he would do just one take of every shot and his films would be absolute garbage. With or without AI, what matters is the commitment, engagement and talent of the creator.

Upscaling footage

I’m not actually a huge fan of high image resolution. I think there’s a lot to be said for smudginess, softness, and a nice bit of film grain or video noise, etc. However, large image scale is clearly desirable in many contexts. Until the advent of AI it wasn’t something that was ever done very well. The AI results speak for themselves - simply incredible.

Restoring colour to footage

Colourisation is usually done to footage that was originally colour but for whatever reason was only preserved as a black & white copy. I happen to know of a more bizarre example: due to an industrial dispute, episodes 2 to 6 of Upstairs Downstairs were unnecessarily shot in black & white. The camera-men disabled the colour tubes on their cameras leaving only the luminance tube operating, which resulted in a black & white image. It has always aggravated me, because I just don’t find these five episodes as engaging as the 63 other episodes of the programme. With AI, this problem could easily be rectified, the episodes restored to how they should have looked. The same is true of series like The Edwardians, where some episodes only exist as black & white copies because the colour mastertapes were junked.

Simulating dead technologies

There have been many camera technologies down the decades, and I believe all of them could be simulated with AI.

Let me give one example. A camera technology that I have great fondness for is colour tubes. The most famous of these cameras, the EMI-2001, debuted in 1968 and was the “workhorse” of the BBC for well over a decade, characterising the look of “70s British TV”. Even as newer cameras made it obsolete, the EMI-2001 remained beloved and BBC engineers cannibalised parts in order to keep an ever smaller number of them going. The last EMI-2001 cameras were finally retired in 1991. The husks of quite a few machines still exist today, but it would be virtually impossible to make one of them operational. The tubes, the crucial element that gathers the image as an electronic signal, aren’t even manufactured any more. It is a dead technology.

But it was a beautiful one. The look of EMI-2001 footage is highly distinctive and gives a story a very particular “feel”.

I have tried - believe me - to make modern digital footage look that way. I have tried both with After Effects filtering and with programming. Nothing comes close.

I believe that AI will manage the task, and this thing from the past will be recovered. Moreover, it will be recovered in high-definition, even 4K. Why not?

Recreating lost footage

We could salvage dramas that are simply lost. In Britain, many TV programmes were junked in the 1960s and ‘70s. For many, complete soundtrack recordings exist. AI could recreate each of those programmes using the soundtrack as the guide as to how the actors should move and look as they speak. Supplementary materials - photos of a set, notes by the costume designer, camera scripts, publicity shots, out-takes, an off-air recording of one particular scene, shots of the actors in other productions or playing similar roles, etc. - could be used to get the AI recreation closer to the original, lost version. Would it ever be exactly the same as the original? Certainly not. But it would be a thousand times closer than what we currently have, which is nothing.

Separating mixed sounds

Dudley Simpson was a composer who wrote and recorded - with the help of his own synthesisers and a small band of multi-instrumentalists - the music for many BBC programmes. A unique talent, Simpson wrote music that is very distinctive and utterly charming. However, the BBC did not keep copies of his music, only the programme soundtracks which have mixed-in dialogue, etc. I once read that Simpson kept his own tapes of his music but lost them all during a house move. As a result, while we can enjoy his music in the programmes, we can hear very little of it by itself. Doubtless Simpson’s work is not alone in this sad fate. With AI, the music could be separated out from the programme soundtracks and finally heard on its own.

Sound resynthesis

A niche area of audio technology is “resynthesis”, where a sound is analysed to produce a unique algorithm - essentially the DNA of the sound - which can then be used to resynthesise the sound at different pitches, for different durations, or with altered characteristics. You could even swap characteristics between sounds - give a trombone the body of a grand piano, etc. Impossible sounds, things we really have never heard before, would become possible.

Despite its great potential, resynthesis has always been a niche thing, for the simple reason that it has never been possible to do it well. One attempt was the Hartmann Neuron, but its capabilities were nowhere near what one would hope for. The technology simply wasn’t there in 2002, because a necessary part of analysing a sound is recognising - as a human would - its characteristics. The Neuron could only guess at the characteristics of what it was “hearing”. AI would be much better at this task.

Trait transposition

A similar thing could be done on the visual side: recognising the elements of an image and recreating them elsewhere. For example, from a city street scene:

• that car

• that truck

• that building

• that traffic light

• that pedestrian

• that group of pedestrians

• this density of traffic

• the general layout of the street

• the weather

• the light conditions (eg. time of day, position of the Sun)

• the visual texture of the recording medium

• that camera move

Element melding

It will also be possible to create elements in the real world and integrate them into the AI work. For example, you could have a real actor give a real performance, then have the AI inject them into a scene. You could even have it change their face or accent. You could take real footage of a setting and have the AI digest that and produce new shots of that setting. You could record a real camera sweep using your smartphone, then tell the AI to reproduce that exact sweep across a digital setting. You could integrate a real matte background you have painted, or a real miniature model you have constructed, or a real puppet you have animated with sticks or strings or servos, etc. In this way, AI imaging could actually reinvigorate traditional animation, sound creation, and techniques of “in-camera” special effects, by providing new ways to integrate them into a drama.

Character creation

I can imagine a software environment for gradually creating characters, over time. This would be visual, aural (their voice) and psychological (their personality). It would encompass their clothes, their build, their face, their tics and mannerisms, how they use their voice… everything. During the process of creating a character, the user could tell the AI to experimentally show him in different situations - eg. at the supermarket checkout, selling a house, breaking up with a lover - which in turn would inspire the user in terms of story, dynamics with other characters, etc. Once the user is satisfied with the character, the software could render him photo-realistically in AI videos - a character performed without the intermediary of an actor to perform him.

Conclusion

As we have seen, AI means that the handling of sound and image can radically change. Creating the raw element - a sound, a scene, even a character - will not be done in the old-fashioned way of creating or finding a physical thing and somehow capturing it in an electronic record, but by conjuring it by verbally instructing software. Then the raw element will be integrated into a digital scene, or created incidentally as a part of the same when it is conjured wholesale. This could be done with a thousand elements that make up a scene, some of them being created individually, some in tens or hundreds. Some will be picked out afterwards for special augmentation, some removed, some multiplied. The process could be endlessly complex or guilt-inducingly simple. But the creator will be driven by his vision and what it uniquely requires, and some elements will be important enough to warrant his attention and others will be incidental and left to the whims of the software. This balance between man and machine will, of course, be critical to whether the result can be considered beautiful art or worthless slop.

In summary, AI tools are incredible not only for their ability to fix or salvage what has been made, but to create from scratch that which has not been made and, indeed, could otherwise never be made, especially by a sole person. If I am right, then how we produce drama will entirely change, and accordingly, how we consume and consider drama will be likewise transformed.