How to Seamlessly Mix Animation with Live Video (no green screen necessary!)

In one of our previous articles, we taught you how to combine stop motion with live video using a green screen. But this time, we'll teach you how to do it without using the chroma key technique at all!

The secret is called masking, and we'll share with you the detailed process on how to do it seamlessly.


So, what's masking?


Masking is one of the earliest practical effects used to combine two scenes.

gif

The term masking comes from practice of covering the lens partially to expose only one side of the film. Once they finish recording the first scene, they would rewind the film, mask the other side of the lens, and start recording on the unexposed part of the film.


These days, you can create a mask in post using either Photoshop, Premiere, or After Effects. And thankfully, you don't need to mask your camera lens to do it!


For this tutorial, we'll focus on using Adobe Premiere. But the concept is the same even when using Photoshop or After Effects.


Step 1: Plan your animation


Masking is an excellent technique to use in certain situations, but it's not a foolproof replacement for a green screen.


This technique only works if your animation and recorded video elements don't "touch" or overlap. The recorded video must always be on one side and the animated clip on the other as you see below.

Looking at our sample clip, you'll see that the action figure stays on one side of the frame, and the video clip where the hand goes in is on the other side. The technique won't work if the action figure goes in front of the box or if the hand touches the toy.


Step 2: Film The Live Video


When filming the video, it's crucial to place your camera on a tripod and ensure it stays in the same position once you start animating.


If you're not using Dragonframe, all you have to do is record your video, switch to your Still Camera Mode and start taking pictures for stop motion.


If you're using Dragonframe, you'll need first to switch your camera to Video Mode. Otherwise, the program cannot record video at all. You also need to make sure you have an SD card in your camera, otherwise you'll see error messages. The app needs it to save a copy of the video in the camera itself.


After you connect your camera to your computer, click Dragonframe's Cinematography tab (the camera icon) on the top right corner of the screen. Look for the Video Capture Mode, and activate it by pressing the big Power Button icon covering the panel.

The Video Capture Mode allows you to change your camera settings within the program. But keep in mind that you can't use slow shutter speed in video.


That's why it's crucial to change the aperture and ISO until you get the correct exposure. Also, make sure that you use the same White Balance setting for both video and photo.

Once you're ready to start, click the red Record button, and Dragonframe will save the footage on your computer.


Remember that your footage will not automatically appear on your animation timeline. You'll need to right-click and select Use as Reference Layer so it appears as an overlay displayed on top of your animation.


Step 3: Export live video footage into your animation timeline


Before you begin your animation, go to the Animation panel and look for the Media Layers under Guide Layers. There you can find the thumbnail of the live video you recorded and adjust its Opacity so you can see the animation you'll be doing below it.


The beauty of using your video as an overlay is that you get to see how your animated clip and your live footage will interact. Dragonframe even slices the video into individual frames to help you better line up your animation with the live clip.


Step 4: Synchronaize your live video and animation with the X-sheet


Short for Exposure sheet (also known as a dope sheet), this tool is crucial for helping you organize how you shoot complex animations, such as combining video and stop motion work.

There are many ways to use the X-sheet, and we will be discussing it on our blog soon. But for now, we'll be teaching you specifically how to use it when working with live video.


The X-sheet's main job is to make sure your stop motion aligns with your main video. Go to your animation timeline and look at the video frames. Now pick the part where your animation needs to interact with your live video.


In our case, we needed our Voodoo Ranger action figure to appear startled as soon as the hand grabbed the beer can. So we look for the part in the video where that action happens and list down the frame number.

We mark the frame number on the X-sheet to remind us how many shots will take until our action figure interacts with a particular section in the live scene. In the case of our sample video, that section starts at Frame 25.


Step 5: Animate the stop motion sequence


For the most part, animating the stop motion part is relatively straightforward. Only this time, you'll need the video frames in the Animation timeline as well as the X-sheet to guide you.


Make sure that you time your movements so your product or character synchronizes with the live video. For our sample animation, it meant that our action figure had to finish walking at precisely the 25th frame. So that means we have to make sure the walking sequence of the action figure fits within 25 frames. Otherwise, the movements in the video and the animation won't be in sync.

Every time you press the Record button on your Dragonframe, the program automatically adds an X to the corresponding frame in your dope sheet. That way, you can keep track of the frames you already shot.


Step 6: Combining your video and animation in Adobe Premiere


First, look for your recorded video in Dragonframe's Tests folder. Alternatively, you can also access your camera's SD card to obtain a copy of the video.


Next, drag it into the second track of Adobe Premiere, then take your animation and load it into the FIRST track in Premiere.

Select the video track. Then go to Effects Controls and click the Free Draw Bezier tool (pen icon) under Opacity.

Draw around the part where you want to keep the video. In our case, we wanted to retain the box of cans because that's where the hand comes in. Once you complete your selection, you'll automatically see the animation under the video track.


Increase the Mask Feather to help you make the edges of the mask disappear.


If the colors or lighting in your video don't match your animation, feel free to color correct it. Click the Color tab and adjust the various exposure parameters until both your clips fit seamlessly.

You'll know you did an excellent job if your final video looks like it was all taken in one shot and not two images composited together.

 

Masking in animation is an advanced technique. But we recommend that you try it early on because it will expand your creative options.

gif

Start with simple concepts to get used to the variables you may encounter while shooting. As you gain more experience, you'll find the process easier to execute.