About AMP Lab        Projects        Downloads        Publications        People        Links

Project - Active Image-Based Rendering 

Contents

 

Team Member

Cha Zhang

czhang@andrew.cmu.edu

Top of this page

 

Motivation

Image-based rendering (IBR) is a hot research area where computer graphics, computer vision and signal processing meet together. Originated from the computer graphics field, much work has been done on how to render virtual images from the captured ones. Representative approaches are lightfield, lumigraph, concentric mosaics, unstructured lumigraph, etc. 

However, much less work has been reported in the capturing aspect. In this project, we want to study how we can capture the scene more efficiently. Our approach is a combination of algorithms in different fields, which is challenging but very  interesting. 

Top of this page

 

System Description

The color consistency criterion  

Active IBR is based on one simple criterion: the color consistency criterion. For Lambertian scenes, color consistency criterion verifies that light rays from the same object surface point should have the same color (intensity). For a non-Lambertian scene but not highly specular, we define the color consistency criterion as: light rays from the same surface point should have the same color, as long as their angles of emission are close enough. 

Estimate the rendering quality 

Assume we have some volumetric model of the scene. For each voxel, we may verify its color consistency as follows: project it to all the images, and see how consistent their colors or intensities are. Since scene may be non-Lambertian, the consistency verification may be performed neighborhood by neighborhood. 

Geometry reconstruction 

Many algorithms exist for reconstructing the volumetric model. Recently the work by Seiz and Dyer on voxel coloring is very interesting and is applied in our system.  

The system working flow 

The active IBR working flow is as follows.  We start the algorithm by capturing an initial set of images uniformly. We then apply voxel coloring algorithm to obtain a 3D voxel model. With the voxel model and the color consistency criterion, we can locate which neighborhood to split with the rendering quality estimator. After the splitting, we may continue capturing new images or applying voxel coloring again for geometry refinement (Since the computation of voxel coloring is heavy, we may prefer to do it for every several capturing steps). The whole process loops until the maximum number of images is reached, or all the images have been color consistent.

Examples 

We experiment active IBR on different setup. In a computer-simulated environment, we implemented a lightfield-like setup, where the capturing cameras are on a plane. In a real system, we use a inward-looking concentric mosaics. There are some video clips in the downloads session that shows the idea.  Two snap shots are shown below: 

       

Top of this page

 

Publications
  • C. Zhang and T. Chen, "Surface Plenoptic Function: Spectral Analysis for Image-Based Rendering", submitted to CVPR2003, Madison, Wisconsin, USA, June, 2003. 
  • C. Zhang and T. Chen, "Surface Plenoptic Function: A Tool For the Sampling Analysis of Image-Based Rendering", submitted to ICASSP2003, Hong Kong, China, April 2003. 
  • C. Zhang and T. Chen, "On Generalized Sampling on Image-Based Rendering Data", ICASSP2003, Hong Kong, China, April 2003. 
  • C. Zhang and T. Chen, "Spectral Analysis for Image-Based Rendering Data", revised and re-submitted to IEEE Trans. on CSVT Special Issue on Image-based Modeling, Rendering and Animation
  • C. Zhang and T. Chen, "Towards Optimal Least Square Filters Using The Eigenfilter  Approach", ICASSP 2002. , Orlando, FL, USA, May 2002.
  • C. Zhang and T. Chen, "Generalized Plenoptic Sampling", Carnegie Mellon Technical Report: AMP01-06. 

Top of this page

 

Downloads
  • Demo video for computer-simulated environment active IBR (download, 18.2 MB). 
  • Demo video for our real capturing system (320x240, 63.5 MB; 160x120, 13.4 MB) 

Top of this page

 

Contact

Any suggestions or comments are welcome. Please send them to Cha Zhang

Top of this page


This website is maintained by Trista Chen
Copyright 200
1 Advanced Multimedia Processing Lab. All rights reserved.
Revised: April 18, 2002 .