iModel: Interactive Co-segmentation for Object of Interest 3D Modeling
People
Adarsh Kowdle, Dhruv Batra, Wen-Chao Chen, Tsuhan Chen
Abstract
We present an interactive system to create 3D models of objects of interest in their natural cluttered environments.
A typical setting for 3D modeling of an object of interest involves capturing images from multiple views in a multi-camera studio with a mono-color screen or structured lighting. This is a tedious process and cannot be applied to a variety of objects. Moreover, general scene reconstruction algorithms fail to focus on the object of interest to the user. In this paper, we use successful ideas from the object cut-out literature, and develop an interactive-cosegmentation-based algorithm that uses scribbles from the user indicating foreground (object to be modeled) and background (clutter) to extract silhouettes of the object of interest from multiple views. Using these silhouettes, and the camera parameters obtained from structure-from-motion, in conjunction with a shape-from-silhouette algorithm we generate a texture-mapped 3D model of the object of interest.
Publications
- Adarsh Kowdle, Dhruv Batra, Wen-Chao Chen and Tsuhan Chen. "iModel: Interactive Co-segmentation for Object of Interest 3D Modeling ", Workshop on Reconstruction and Modeling of Large-Scale 3D Virtual Environments, European Conference on Computer Vision, 2010 (ECCV '10). [pdf] [slides]
- Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jeibo Luo and Tsuhan Chen. "iCoseg: Interactive Co-segmentation with Intelligent Scribble Guidance", Computer Vision and Pattern Recognition, 2010 (CVPR '10). [pdf] [Project Page]
Video showing 3D visualization of the results
Demo
- Adarsh Kowdle, Haochen Liu, ShaoYou Hsu, Jason Lew, Charvi Puri, Dhruv Batra, Tsuhan Chen. "iModel: Object of Interest 3D Modeling via Interactive Co-segmentation on a Mobile Device;, Computer Vision and Pattern Recognition, 2012 (CVPR '12).
We develop an iOS application allowing the user to obtain the 3D model of their object of interest by capturing a video around the object in its natural environemnt. The video is sent to a remote server along with simple scribbles on just one image. The server then performs the computation and returns to the client iOS device the multi-view co-segmentation of the object of interest and the 3D model of the object.
Video showing a demo of iModel on an iPhone
3D printing objects of interest
Our approach lends itself to a very practical application of 3D printing an object of interest as shown below.
Cornell Multiview Dataset
We make publicly available two sets of multiview datasets as the Cornell Multiview Dataset.
- Object_Multiview: Images of an 'object of interest' captured from multiple viewpoints around the object, with the goal of co-segmentation and obtaining a 3D model of the object of interest
- Scene_Multiview: Images of a scene captured from multiple viewpoints with the goal of dense depth estimation of the scene.
If you use this dataset for research purposes, please cite us in any resulting publication. The following are the relevant publications,
Object Multiview:
- Adarsh Kowdle, Dhruv Batra, Wen-Chao Chen and Tsuhan Chen. "iModel: Interactive Co-segmentation for Object of Interest 3D Modeling ", RMLE Workshop at the European Conference on Computer Vision, 2010 (ECCV '10).
- Dhruv Batra, Adarsh Kowdle, Devi Parikh, Jeibo Luo and Tsuhan Chen. "Interactively Co-segmentating Topically Related Images with Intelligent Scribble Guidance", International Journal of Computer Vision, 2011 (IJCV '11).
Scene Multiview:
- Adarsh Kowdle, Yao-Jen Chang, Andrew Gallagher and Tsuhan Chen. "Active Learning for Piecewise Planar 3D Reconstruction", Computer Vision and Pattern Recognition, 2011 (CVPR '11).
- Adarsh Kowdle, Yao-Jen Chang, Dhruv Batra and Tsuhan Chen. "Scribble Based Interactive 3D Reconstruction via Scene Co-segmentation", International Conference on Image Processing, 2011 (ICIP '11)
Software: iScribble
We have made the graphical user interface developed by us available to the community. We feel that this GUI is a powerful yet simple tool which can be easily integrated with Matlab. Please feel free to use it and give your feedback.
You can download the interface from the iScribble project page.