Advanced Multimedia Processing Lab -- Projects -- IBRL

            About AMP Lab        Projects        Downloads        Publications        People        Links

Project - Image-Based Relighting System

Contents
Team Member

HyunJung Kate Shim

hjs@andrew.cmu.edu

Top of this page

                                  

Motivation and Goal

Image-based relighting (IBL) has become a popular research topic in both computer graphics and signal processing. 

With image-based relighting, one can render realistic relit images of a scene without prior knowledge of object geometry in the scene. However, traditional IBL methods require a large number of basis images, each corresponding to a lighting pattern, to estimate a 2D light mapping function from each pixel of the scene image. 

 In this project, we develop a statistical approach to estimating a 2D light mapping function, an SRF (Surface Reflectance Function), with  fewer input images than previous IBL work. An SRF is defined as a weighting function that describes how each point light sources contributes for a particular pixel value on the image. By having an SRF, therefore, we can synthesize an image under arbitrary lighting condition. Our goal is to minimize input images to estimate an SRF while we maximize the quality of relit images. This work can be extended to the environment matting.

Top of this page

System Description

Our relighting system consists of single camera (canon G5) , 17" CRT monitor (DEL 17") and arbitrary objects. The wire frame outside of the object supports the camera and the object. The frame is necessary to maintain the relative position between the object and the camera. In order to control the lighting environment, we use the black cloth to cover the entire setup while the experiment is running.

We implement the program to control the camera with following parameters; shutter speed, aperture value, ISO speed, image size, image quality, flash, and white balance. We combine the pan-tilt system as well for future work. With specified camera parameters by an user, the camera starts to capture images under the sequence of lighting patterns

 

The lighting patterns are designed by the analysis of SRFs, which describe how a pixel on the image is illuminated by each point light source. Above figure shows the definition of an SRF (Surface Reflectance Function). The illumination of a pixel value reflected by the surface is recorded on the image plane by taking the inner product of an SRF and a lighting pattern. We collect the data of SRFs from synthetic images and analyze them by Principle Component Analysis (PCA). PCA results provide us the most representative basis functions of SRFs statistics. Derived basis functions is shifted and scaled into the range of image. Designed patterns are optimal to reconstruct SRFs in mean square error sense.

 

Above image shows a relighting result from synthetic data. The first image is a rendered image by ray tracing algorithm (POV-RAY) and the second image is a relit image using only fifteen input images.

Above figure is a relighting result of the real objects. The first image is a real image and the second is a relit image using only twenty training images. An applied lighting pattern in above examples does not belong to training images.

Demo file (.m1v)

This is our relighting demo. This video file contains relit images with corresponding lighting conditions. We have used 20 input images to generate above relit images.

Top of this page

Publications
 

Top of this page

Contact

Any suggestions or comments are welcome. Please send them to Kate Shim 

Top of this page


This website is maintained by Wende Zhang
Copyright @ 2001 Advanced Multimedia Processing Lab. All rights reserved.
Revised: April 10, 2005 .