Project - Active Image-Based Rendering
Image-based rendering (IBR) is a hot research area where computer graphics, computer vision and signal processing meet together. Originated from the computer graphics field, much work has been done on how to render virtual images from the captured ones. Representative approaches are lightfield, lumigraph, concentric mosaics, unstructured lumigraph, etc.
However, much less work has been reported in the capturing aspect. In this project, we want to study how we can capture the scene more efficiently. Our approach is a combination of algorithms in different fields, which is challenging but very interesting.
The color consistency criterion
Active IBR is based on one simple criterion: the color consistency criterion. For Lambertian scenes, color consistency criterion verifies that light rays from the same object surface point should have the same color (intensity). For a non-Lambertian scene but not highly specular, we define the color consistency criterion as: light rays from the same surface point should have the same color, as long as their angles of emission are close enough.
Estimate the rendering quality
Assume we have some volumetric model of the scene. For each voxel, we may verify its color consistency as follows: project it to all the images, and see how consistent their colors or intensities are. Since scene may be non-Lambertian, the consistency verification may be performed neighborhood by neighborhood.
Many algorithms exist for reconstructing the volumetric model. Recently the work by Seiz and Dyer on voxel coloring is very interesting and is applied in our system.
The system working flow
The active IBR working flow is as follows. We start the algorithm by capturing an initial set of images uniformly. We then apply voxel coloring algorithm to obtain a 3D voxel model. With the voxel model and the color consistency criterion, we can locate which neighborhood to split with the rendering quality estimator. After the splitting, we may continue capturing new images or applying voxel coloring again for geometry refinement (Since the computation of voxel coloring is heavy, we may prefer to do it for every several capturing steps). The whole process loops until the maximum number of images is reached, or all the images have been color consistent.
We experiment active IBR on different setup. In a computer-simulated environment, we implemented a lightfield-like setup, where the capturing cameras are on a plane. In a real system, we use a inward-looking concentric mosaics. There are some video clips in the downloads session that shows the idea. Two snap shots are shown below:
Any suggestions or comments are welcome. Please send them to Cha Zhang.