Intro

Combining computer generated (CG) characters with live-action footage can be a very rewarding experience. It has traditionaly been very difficult to make the digital character realistically seem to be a part of the real world. Animation:Master offers a number of solutions to make the task relatively easy.

One of the challenges in making a CG character part of a real background is making the character cast realistic shadows onto his surroundings. After all, the light source that illuminated your scene never actually comes in contact with your synthetic computer model. In a nutshell, you can overcome this obstacle by building rough geometry that mirrors your real-world location and cast shadows onto this geometry. This tutorial is designed to take you step-by-step through two solutions that will allow you to combine your own characters with any location that you shoot.

Location, location, location

Before you even touch the computer, your first task is to find a location. Although it’s not mandatory, it’s advisable to find an area that has a relatively flat ground. This will make your job much easier, especially when trying to recreate your scene as a computer model, not to mention it’s far easier to animate a character on a level ground.

Next you will have to decide where you want to put your camera and also the direction you want it to face. Framing your shot can be a little tricky because you can’t see what will end up being the focus of the shot – the character. You don’t want to take your footage back into the computer and find out that your character’s head is going to be cut off. After setting up my camera, I shot myself walking down the sidewalk. Since I’m taller than my CG character, I know that if I am in the shot, he definitely will be. This will also serve as good reference for how and where your character will cast shadows.

Now you have to measure out your location. Using a tape measure, measure (a) the distance between the ground and your camera, (b) the distance between the camera and any objects upon which your character may cast shadows, and (c) the distance relationship between objects. Be conservative here – the more accurate you are with your measurements, the quicker your task will go. Basically, you want to end up with an overhead "floorplan" of your location so that you can easily recreate it in the computer.

Shadow-catching geometry

Open up A:M, make a new project and select the Project itself from the Project Workspace (PW). In the project’s Properties window, make sure the frame rate is set to 30 frames per second (FPS) if you use NTSC video (primarily in the US) and 25 fps if you use the PAL format (primarily European). This is because you will want your project to be the same frame rate as your original footage. For this tutorial, I will use NTSC footage so any reference I make to 30fps can be translated to 25fps for those using a PAL system.

Now you have to build a rough version of your scene in A:M. Luckily, A:M allows you to use real world measurements when modeling so if your measurements are in feet and inches, your models can as well. If your measurements are in meters and centimeters, you can model in metric units. [hint – the computer doesn’t care which units your models are in – it only cares about their relative size to eachother] These models do not have to be 100% accurate as their only function is to catch your model’s shadow. Feel free to model very roughly and refine as the need arises.

Next you have to make a choreography and drop your shadow-catching location model(s) into it. Check the "Shortcut to…" your model in the choreography and make sure that "Active," and "Receive Shadows" are checked and "Flat Shaded," "Cast Shadows," and "Cast Reflections" are turned off in the General tab of the Properties window. Now you have a rough, shadow-catching version of your location in the computer. What’s missing besides the character? The camera. Using your ‘floorplan,’ move the camera in the choreography to the same place it was in the real location. Make sure you move it off the ground, to the same height it was during the shoot.

Matching the camera view

After digitizing your footage (making it an AVI, MOV or sequential images), right-click (Command-click on the Mac) your "Shortcut to Camera 1" in the project workspace (PW) and select "New Rotoscope." A box will appear with all of the bitmap images and AVI/MOV clips in it. Click the "browse" button and select your digitized footage. If you look through the camera view in your choreography, you should see your footage as a backdrop. Chances are that the real set is not matching your virtual set. This is because the angle of the virtual camera and it’s aperture are probably set differently than your real one. From the camera’s point of view, you can change these attributes using the following keyboard commands:

Rotate camera - Click "t" hold down <shift> and click-drag your mouse
Tilt camera – Click "t" hold down <Ctrl> and click-drag your mouse
Change focal length – Click "z" hold down <shift> and click-drag your mouse
These attributes and others can also be changed numerically in the camera shortcut’s Properties window.

Assuming that your measurements are correct, you should find that your footage will match your geometry after some trial and error. If some things are a little off, feel free to change the model or the model’s placement in the choreography. I find it much easier to change these things within a second choreography window so I can see it update from the camera view in real time.

Solution 1: Front Projection Mapping

Now you need this geometry to accept your footage as an image map. Quite literally, FPM projects your footage onto geometry like a slide projector. Of course, this map has to be applied or projected from the camera’s point of view. A:M’s front projection mapping makes this incerdibly easy. Just select the shortcut to your scene model(s) and click the checkbox labeled "Front Projection Target" in the General tab of the model’s Properties. That tells A:M to use your camera’s rotoscope as a front projection map and to map it onto the model(s). If you render now from your camera’s POV, you should see your scene with discolored objects where your footage is projected onto the geometry. In the color tab of the model’s properties box, change the falloff number to match your footage so that there are no seams. This will require some trial and error but as a general rule, if it’s too dark, lower the falloff. If it’s too light, increase the falloff. Try to use the highest number you can while still maintaining a seamless image.

The footage that I brought in was 776 frames and I knew it was too long. So that I would have room to breathe, I shot extra footage before and after the main background action – a bus driving by. So in A:M, I selected the footage from the Images/Animation folder in the Project Workspace (PW) and, after making sure it was set to 30fps, cropped the movie to 501 frames (start at frame 30, end on 530). Now I have about 16.5 seconds of footage over which I can animate.

Lighting

Now you need to drop your character into the choreography. Put him into view of the camera, scale him to the proper size and do another test render. Assuming your camera is set to render shadows, your character should be casting shadows onto the geometry. Unfortunately, the shadow is probably facing a different direction than the rest of the elements in the shot. Your task now is to try to re-create the lighting of the shoot so that your character doesn’t seem out of place. In my footage, I used an outdoor scene therefore I use a sun type light in the project. Lighting is quite a specialized task and requires years of learning and practice to get right. For this tutorial, I just "eye-balled it" using the information that is available on the original footage. For a totally accurate lighting model, there are some great references available and I will try to list some of the best in the ‘recommended reading’ section of this tut.

Using a sun light, I found that the digital shadows were quite a bit darker than those of the real objects in the scene. Because I had footage of myself casting a shadow on the same sidewalk, I had great reference for how opaque the shadow should be. Since I can’t modify the opacity of the shadow itself, I have to change the attributes of the object it’s being cast upon. In this case, I raised the ambience of the shadow-catching geometry until I had a good match.

Rendering

Now you get to render your footage and see how it worked out. Chances are, you’ll be pretty close at this point but your scene may require some tweaking to get it just right. How much tweaking really depends on the complexities of your shot.

One thing you may notice about your video footage is the way horizontal lines appear across moving objects. This is normal. Remember I said video runs at 30 fps? I lied; it’s actually 60 half frames per second. Those hozitonal lines are the result of seeing two half frames woven together, or "interlaced." When viewed on a television in full motion, this interlacing is not noticeable. But if your animated character is rendered at 30 full frames per second and is layed on top of video that is 60 inherantly split images, your character will look out of place. To fix this, make sure in your final render options, you select "Field Render" from the Video tab. This will interlace your CG character into the live action background. The field order (odd or even) you will need to use depends on your captured video footage, details of which can be learned from your video card or camera’s documentation. If you select the wrong one, you will notice it in your render – since there are only two choices, the one that looks blatently wrong is the wrong choice. You will also want to check the box for "NTSC Safe Colors" so that any saturated colors in your CG elements that lie outside of the limited color space that televisions allow will be muted so that they don’t seem out of place.

Now render your animation to the specifications suggested by your video card manufacturer and you should be able to watch your character interacting with the location you shot. The success of the illusion depends on a variety of circumstances and there are few hard-and-fast rules when it comes to matching virtual and practical elements.

Solution 2: Multiplane

Two new features of Animation:Master (as of v8.0) are Multiplane and shadow-only geometry. Multiplane is a powerful compositing suite included within A:M. You can render your character and the shadow he casts as two separate animation sequences and combine them with the original footage within A:M. This gives you a bit more control over the look of the shadow and also allows for some very complex compositing scenarios.

First select your original footage in the "Shortcut to camera’s" rotoscope in the PW and make it invisible by unchecking the Visible checkbox in the Properties window. Now select your scene model(s) and make them inactive by unchecking the Active checkbox in the properties window [caveat: be sure to do this on the first frame of your choreography. The Active checkbox is a keyframable item which means you can make your model switch between active and inactive throughout an animation. If you make him inactive on a frame after the first one, he will be active until that frame when he will disappear.]

Now you will want to render your animation of just the character (sans background) with an alpha channel (i.e. – check the Alpha box in the Buffers fieldin the Render Output tab). See Rendering above for field settings. Then, you will need to render out just the shadows cast by the character as it’s own layer. To do this, go back into your choreography and make all of your models Active again. In the ‘Shortcut to’ your shadow-catching model’s properties window, check "Shadow Only" and this time, render with "Shadow Buffer Only" selected in the Output tab of your Render Panel (make sure you rename your animation sequence so as to not overwrite your character animation!). This will render a solid black frame with an 8-bit alpha channel which acts as a ‘travelling matte’ to knock the shadow’s shape out of it.

Now make a new choreography and you can import your background footage, the shadow pass and the character pass as three rotoscopes for a single camera. If you import them in that order, you will see the equivilent of the FPM technique, only the shadow will be solid black (the alpha channel automatically knocks out the shadow shape). In that rotoscope’s (layer’s) properties box, set the transparency of the layer to a number less than 100% to match your footage and render away. [hint – if you didn’t import them in the right order, feel free to drag and drop the rotoscopes in the PW so that the footage is the first roto (bottom) and the character is the last (top)] Rendering this animation will provide similar results to the FPM method.

Now that you understand the thoery of front projection mapping and multi-plane rendering/compositing within Animation:Master, you can experiment with your own footage and characters. You will find that virtually any scenario can be accomplished with proper planning and creative compositing solutions.

-- Ed

-----