Capturing an Avatar

Chroma Keying

We look at the individual components of each pixel (red, green, and blue) and calculate an alpha (transparency) value for that pixel based on these other values.

Based on Luminance and Chrominance
Using YUV color-encoding method.

Luminance (Y) is made up of a certain percentage of each color component:

    RED --- 0.299
    GREEN --- 0.587
    BLUE --- 0.114

Multiplying each color component of a pixel by its respective percentage and adding them together gives the luminance for that pixel.

We can then use the values:

Red - Luminance --- (R - Y) or (V)
Blue - Luminance --- (B - Y) or (U)

to determine which pixels to drop out.

If the (R - Y) value of a given pixel is within a certain range, and its (B - Y) value is within another given range, then the pixel can be dropped out.

We can take this a step further and compare the (R - Y) and (B - Y) values of the pixel to their upper and lower bounds independantly:

    (R - Y) - Rmin > 0
    Rmax - (R - Y) > 0
    (B - Y) - Bmin > 0
    Bmax - (B - Y) > 0

Each of these comparisons yields a result in the range 0 to 1. These four results are then multiplied together, producing another number in the range 0 to 1, which can then be used to determine what percentage of the pixel is transparent. This way the pixels have a variable amount of transparency rather than being either fully transparent or fully opaque. This makes the edges softer and less jagged.

We can see how this works by plotting each pixel on a graph. The (B - Y) is plotted on the horizontal axis, and the (R - Y) on the vertical, similar to a vector scope. We can also draw the bounding box created by the upper and lower limits of (R - Y) and (B - Y) values. The pixels that lie within this bounding box on the graph are the ones that will be made transparent.

Diagram

Diagram



Recording an Avatar

Currently things are configured to work properly on Torrance in room 3036.

Camera Set-up

Connect the hi-8 camera's Video Out to the Composit Video In on the Impact. (You could also loop the signal through a video monitor.) The camera should be placed at eye-level on a tripod about 12 or 15 feet from the bluescreen (pretty much as far as you can get it in the current set-up.) The camera should be rotated -90 degrees, so that the person is horizontal in the image with their head to the left of the screen. The reason for this is to capture the highest resolution possible. Since the texture map that we will ultimately use needs to be a power of 2, rotating the camera allows us to get an image of 512x256 pixels, as opposed to 256x128.

Software Set-up

There are two programs, VAgui and VAip, in ~insley/VidAvLib/capturing. Copy them to a directory in /usr/tmp/. This is so that when you are recording it doesn't have to write across the network. Otherwise you will be likely to drop frames in the recording process.

Sometimes the video daemon gets confused. To make sure that it is working properly, start up vcp. If it opens up all right, try opening Utilities->Live Video Input. If that seems to be working properly, exit Video In and vcp, you're ready to proceed. If either Video In or vcp itself failed, reboot the machine to reset the video daemon.

Start up VAgui and run it in the background (VAgui &)

Diagram

Start up VAip (make sure that it is run in the foreground.)

Once VAip has been started and all of the graphics windows are updating, you need to make a dummy recording to finish setting up the recording process. Press the Record button in the VAgui window. The graphics windows will temporarily stop updating. The winterm window will prompt you to press "Enter" on the keyboard. Do so with the cursor in the winterm window. You will then be prompted to press the Continue button in the VAgui window. The graphics windows should resume updating, and you're ready to drop out the background.

Selecting the Capture Area

Diagram

The VideoAvatar image processor window shows the full image which will be recorded. The area within the red box represents the area which will actually be used for the VideoAvatar. This area can be adjusted. When the cursor is inside the box, the box will turn BLUE, indicating that it can be selected, using the left mouse button. When it is selected, it becomes GREEN, indicating that it can be moved. It should be positioned so that the blue of the backdrop fills the capture area, with the person's feet as close to the right side as possible (or to the bottom in the VideoAvatar processed image window.) The polygon which these textures are eventually mapped on to are a fixed size of 6.5 feet. So if you want the avatar to be approximately to scale, you should leave the appropriate amount of space above the person's head. You may also need to make some adjustment from the camera to do this properly.

Selecting the Drop-out Area

In the VAgui window, the Soft Key slider controls the softness of the edges of the dropped out areas. All the way to the left means that there is no differentiation in the alpha values of the pixels. All the way to the right means that all pixels are either fully opaque of fully transparent. So as this value increases the edges get sharper (jagged), as it decreases the edges get softer.

As described the the Chroma Keying section above, in the VideoAvatar vector window the bounding box of the (R - Y) and (B - Y) minimum and maximum values determine which pixels get dropped out. These values can be adjusted individually or all together. Just as the capture area in the VideoAvatar processed image window, the box (or portion of the box) will turn BLUE when it can be selected, and GREEN when it is selected and can be adjusted.

Making a Recording

Once the background is dropped out correctly you are now ready to actually record a movie file. The full frame, unprocessed images are actually recorded, and the information about which portion of the image to use and how to drop out the background is written to a file. Both of these files are read in by the CAVE program, and the information used to process the images at that time.

When you are ready to record, press the Record button in the VAgui window. Again the graphics windows will stop updating, and this time you will be prompted to enter a BaseName. This BaseName will be used for the movie file and the chroma keying information file (BaseName.vamv and BaseName.vachroma, respectively.) Once you've entered the BaseName you will again be asked to press "Enter" on the keyboard. IMPORTANT: You should begin recording when the person on the turntable is facing to the left from the camera's point of view. After the person has turned out complete revolution (15 seconds) the recording is complete and you will once again be prompted to press the Continue button in the VAgui window. Repeat to capture another avatar, or kill the process from the winterm window to exit.

Don't forget to copy your files to your home directory, or where ever you plan to store them, as the automatic purge of files in usr/tmp is once again working.



Back to VideoAvatar