 |
About AMP Lab Projects Downloads Publications People Links
Project - Active Image-Based Rendering
Top of this page
Image-based rendering (IBR) is a hot research area where computer graphics, computer vision and signal processing meet together. Originated from the computer graphics field, much work has been done on how to render virtual images from the captured ones. Representative approaches are lightfield, lumigraph, concentric mosaics, unstructured lumigraph, etc.
However, much less work has been reported in the capturing aspect. In this project, we want to study how we can capture the scene more efficiently. Our approach is a combination of algorithms in different fields, which is challenging but very interesting.
The color consistency criterion
Active IBR is based on one simple criterion: the color consistency criterion. For Lambertian scenes, color consistency criterion verifies that light rays from the same object surface point should have the same color (intensity). For a non-Lambertian scene but not highly specular, we define the color consistency criterion as: light rays from the same surface point should have the same color, as long as their angles of emission are close enough.
Estimate the rendering quality
Assume we have some volumetric model of the scene. For each voxel, we may verify its color consistency as follows: project it to all the images, and see how consistent their colors or intensities are. Since scene may be non-Lambertian, the consistency verification may be performed neighborhood by neighborhood.
Geometry reconstruction
Many algorithms exist for reconstructing the volumetric model. Recently the work by Seiz and Dyer on voxel coloring is very interesting and is applied in our system.
The system working flow
The active IBR working flow is as follows. We start the algorithm by capturing an initial set of images uniformly. We then apply voxel coloring algorithm to obtain a 3D voxel model. With the voxel model and the color consistency criterion, we can locate which neighborhood to split with the rendering quality estimator. After the splitting, we may continue capturing new images or applying voxel coloring again for geometry refinement (Since the computation of voxel coloring is heavy, we may prefer to do it for every several capturing steps). The whole process loops until the maximum number of images is reached, or all the images have been color consistent.
Examples
We experiment active IBR on different setup. In a computer-simulated environment, we implemented a lightfield-like setup, where the capturing cameras are on a plane. In a real system, we use a inward-looking concentric mosaics. There are some video clips in the downloads session that shows the idea. Two snap shots are shown below:
| Publications |
- C. Zhang and T. Chen, "Surface Plenoptic Function: Spectral Analysis for Image-Based Rendering", submitted to CVPR2003, Madison, Wisconsin, USA, June, 2003.
- C. Zhang and T. Chen, "Surface Plenoptic Function: A Tool For the Sampling Analysis of Image-Based Rendering", submitted to ICASSP2003, Hong Kong, China, April 2003.
- C. Zhang and T. Chen, "On Generalized Sampling on Image-Based Rendering Data", ICASSP2003, Hong Kong, China, April 2003.
- C. Zhang and T. Chen, "Spectral Analysis for Image-Based Rendering Data", revised and re-submitted to IEEE Trans. on CSVT Special Issue on Image-based Modeling, Rendering and Animation.
- C. Zhang and T. Chen, "Towards Optimal Least Square Filters Using The Eigenfilter Approach", ICASSP 2002. , Orlando, FL, USA, May 2002.
- C. Zhang and T. Chen, "Generalized Plenoptic Sampling", Carnegie Mellon Technical Report: AMP01-06.
- Demo video for computer-simulated environment active IBR (download, 18.2 MB).
- Demo video for our real capturing system (320x240, 63.5 MB; 160x120, 13.4 MB)
Any suggestions or comments are welcome. Please send them to Cha Zhang.