About AMP Lab        Projects        Downloads        Publications        People        Links

Project - Face Tracking

• Team member

• Goal

• System Description  

• Download  

• Our work is used by...

• Publications

• Contact

Top of this page

Face tracking can serve as a front end to further analysis modules, such as face recognition, face expression analysis, gaze tracking and lip-reading, to name a few. Face tracking is also a core component to enable the computer to "see" the computer user in a Human-Computer Interface system.

We want to build a robust and accurate face tracking system running in real-time, as shown in the Quicktime demo video (Realvideo encoded):

Also we want to extend current single face tracking system to track multiple faces, even when there's occlusion between faces. The image below shows some frames extracted from a real-time video sequence (Realvideo encoded):

Top of this page

The face tracking system is based on the stochastic color model and the deformable template.

• Color model: 

  • We use Gaussian Mixture Model (GMM) to model the color distribution of the face region and the background region:

• Then for each pixel, calculate the log likelihood ratio:
  

• Deformable template

  • We use a logrithmic search to deform the template and fit it with the face optimally, based on the target function we just calculated above:

  

• Constraint based multiple face tracking

  • We expand the algorithm to multiple face tracking based on constraints on the speed and size of the faces:

Top of this page

We build a face tracking toolkit based the algorithm described above. The toolkit is pre-compiled as a dynamic link library (.dll) file on the Windows platform, and a static library on the Linux platform.

Here's some description about the library:

This Face Tracking SDK is a dynamic link library (dll) written in C++. You can link the library to your own program and call the functions provided by the library to perform face tracking in a color image sequence. The core function of the library is FtTrackNextFrame (RECT &FaceRect, unsigned char *ImgData), in which you need to tell the function the location of the face in the previous image frame and provide the image data buffer. The function will update the face location parameters using the information in the current image frame. Note that the face location in the previous frame is stored in the variable FaceRect of type RECT, and the function will update the content of this variable. The pointer ImgData points to a memory buffer containing one image frame in the format of RGBRGB…, with each pixel taking up three bytes.

To use the library, you first need to include the header file named FaceTrack.h into your program. Then you can call the functions provided by the library in your own program. When your program runs, it will dynamically link to the library which is stored at the same directory or in a directory in the system path. 

In your program, you first need to create an object of the type CfaceTrack. It has three main member functions for you to use:

• FtInitialization, which does some initialization work such as allocating memories, creating a binary mask, etc. 

• FtTraining, which calculate the color distribution of the face pixels, and of the background pixels. This distribution information will be used later in the FtTrackNextFrame function to update the face location.

• FtTrackNextFrame, which uses the face location in the previous frame, the image frame data as the input, and update the face location parameters. 

Here’s an example program segment:

{
    CfaceTrack   FTrack;             /*define a face tracking object*/
    FTrack.FtInitialization (640, 480);         /* set the image size to 640x480 */
    unsigned char ImgData = ReadAFrame();         /* read a frame from image sequence
                                     this ReadAFrame function is not defined by the library and should be implementd by the user */
    RECT FaceRect = GetFacePosition ();         /* get face position, only needed at the very first frame 
                                    this GetFacePosition function is not defined by the library and should be implementd by the user */
    FTrack.FtTraining (FaceRect, ImgData);  /* train classifier using current frame */
    BOOL bSuccess =  TRUE;

    while (bSuccess && !IsSequenceEmpty()){
         ImgData = ReadAFrame();  /* read a new frame */
         bSuccess = FTrack.FtTrackNextFrame (FaceRect, ImgData);/* update the parameters */
         if (bSuccess){
                /* TODO: user can add the any other functions here with the given face location*/
         }
    }
}

Download the library.

Top of this page

• ARDA http://www.ic-arda.org/

• GM http://gm.web.cmu.edu/

• Intel http://www.intel.com/

• Microsoft http://www.microsoft.com/

• Nursebot http://www.cs.cmu.edu/~nursebot/

• NSA http://www.nsa.gov/

Top of this page

• F. J. Huang and T. Chen, "Tracking of Multiple Faces for Human-Computer Interfaces and Virtual Environments", IEEE Intl. Conf. on Multimedia and Expo., New York, July 2000

Top of this page

Any suggestions or comments are welcome. Please send them to Fu Jie Huang. 

Top of this page