About AMP Lab Projects Downloads Publications People Links
Office: Porter Hall B10
Lab: Porter Hall B42
Department of ECE, Carnegie Mellon University,
5000 Forbes Avenue, Pittsburgh,
[Research Interests] [Projects] [Publications]
The Mobile Camera Array
Dynamic image-based rendering (DIBR) has been an extremely challenging research topic due to the huge amount of data required. This work tried to build a camera array to capture dynamic scenes and render them in real time. Meanwhile, the cameras are mobile such that they can reconfigure themselves automatically according to the scene content in order to achieve best rendering quality.
Non-Uniform Sampling of Image-Based Rendering and Active Image-Based Rendering
If we take pictures of a natural scene from different directions, we get different pictures. Some are simple, some are complex. Some changes fast when the camera moves a little bit, some changes slowly. Some are important to the user, some are less important. In most cases, we may want to take more pictures from one direction than the others. The non-uniform sampling of image-based rendering has the same flavor. More images are taken when the estimated rendering quality is bad. Other criteria, such as user preference, can be seamlessly integrated into the sampling process.
Spectral Analysis and hexagonal sampling for Image-Based Rendering
Image-based rendering often requires many images in order to avoid aliasing or ghosting effects. The sampling analysis of IBR is thus very important. We study the Fourier spectrum of IBR through a novel method. It is capable to analyze scenes with non-Lambertian surfaces and occlusions, as well as generic IBR representations such as concentric mosaics. We also work on generalized sampling, which view the IBR sampling as a multi-dimensional sampling problem, and apply hexagonal or quincunx sampling on IBR.
3D Model Retrieval
3D scene/object browsing is becoming more and more popular as it engages people with much richer experiences than 2D images. As it is easy to build 3D models from real scene/object today, more and more 3D models are available over the Internet. The growing size of available 3D models makes 3D model retrieval very important. This work focuses on extracting features that well represent 3D models and combining these features in order to get a better retrieval performance.
Active Learning in Content-Based Information Retrieval
Hidden annotation has been proved to be a powerful tool to bridge the gap between low-level features and high-level semantic meanings in a retrieval system. We propose to use active learning to improve the efficiency of hidden annotation. We show that with our algorithm, the performance of the retrieval system improves rapidly along with the number of models that have been annotated. In addition, in all the experiments our algorithm outperforms the random sampling algorithm that picks models randomly to give annotation.
Compression, Real-Time Rendering and Streaming of Image-Based Rendering
Image-based rendering has attracted much attention recently in realistic scene/object representation. By taking thousands of pictures for the real scene/object, image-based rendering can render 3D complex scene/object very realistically yet fast. As many data are involved in image-based rendering, compression is essential through the birth of IBR. We propose some new algorithms to compress IBR at very high compression ratio. The compressed bitstream is random-accessible so that real-time rendering is possible with only the compressed bitstream held in the memory. Moreover, we implemented the world's first application to stream IBR over the bandwidth limited Internent.
Top of this page
The Mobile Camera Array
We build a mobile camera array consisting of 48 network cameras. Virtual views can be rendered from the 48 cameras in real time (5-10 fps). Moreover, a unique feature of our camera array is that the cameras are mobile. Each camera can move horizontally, as well as pan. This mobile camera array opens a wide area of novel research topics.
See the mobile camera array project page for more information!
Active Image-Based Rendering
We build a system for capturing Image-based rendering scenes intelligently. The system first use a simple algorithm to estimate the rendering quality of each neighborhood. Then it decides the position where more images need to be taken. It takes more images, and then go back to to the first step to find the next position. This iteration runs until the maximum number if images reaches. The result is a non-uniformly sampled IBR.
Please refer to the active image-based rendering project page for more details.
Spectral Analysis and Sampling for Image-Based
Image-based rendering has become a very active research area in recent years. The sampling problem for IBR has not been completely solved. We study the the spectral analysis and sampling of IBR. In terms of spectral analysis, we obtain more general results than previous work by using the surface plenoptic function. After we get the spectrum, generalized sampling for high-dimensional data is used to pack the data in the frequency domain.
For more detailed description,please see the IBR sampling project page.
Learning in Content-Based Information Retrieval
We propose to use active learning to improve the efficiency of hidden annotation. Active learning has been studied in the machine learning literature. For many types of machine learning algorithms, one can find the statistically "optimal" way to select the training data. This was given the name active learning. Although in traditional machine learning research, the learner typically works as the recipient of data to do training, active learning enables the learner to use his own ability to respond, to collect data, and to influence the world he is trying to understand. To be more precise, what we are interested here is a specific form of active learning, i.e., selective sampling. The goal of selective sampling is to reduce the number of training examples that need to be labeled by examining unlabeled examples and selecting the most informative ones for the human to annotate.
For more details on the project, please see the active learning project page.
3D Model Retrieval
In our system, we propose a new set of features that view the 3D model as a solid binary region. Ten features such as volume-surface ratio, aspect ratio, moment invariants and Fourier transform coefficients are extracted. In the current version, we simply use Euclidean distance to measure the similarity between two models after the features are normalized.
For more details on the project, please see the 3D model retrieval project page.
Top of this page
Journal Papers and Book Chapters: