The goal of this project is to generate clean, accurate and reliable 3D drawings from multiview image data. Our output representation is a 3D graph representing the geometry and the organization of the scene, and can be regarded as a 3D version of architectural drawings and blueprints.
The input can be a sequence of either video frames or discrete images. If the calibration is not already available, we start by calibrating the input sequence to subpixel accuracy, using the GPUSIFT-based bundle adjustment package included in VisualSFM.
Then we extract image curve fragments using the method of Guo et. al. outlined in , since these image curve fragments typically have more accurate localization and orientations than other existing curve extraction algorithms. You can view the results of this algorithm on our image sequences below.
Using these image curves, we obtain many short 3D curve segments using the hypothesis testing framework of Fabbri and Kimia outlined in , together with our own hypothesis confirmation criteria. Briefly, our method relies on matching pairs of image curves with sufficient epipolar overlap in order to hypothesize the geometry of a 3D curve that corresponds to both of these image curves. Hypotheses are backprojected onto different views and verified using image edge structures. An edge is said to be supporting a hypothesis if it is close enough to the backprojection of the hypothesis, and if its orientation matches the backprojected tangent.
In order to confirm a hypothesis as a real 3D curve, the measures we use are:
We alternate between all the views and reconstruct as many curves as we can. During this process, a lot of redundant reconstructions are made. We use the sample ordering on image curves to enforce an alignment between curve segments reconstructed from the same image curves, allowing us to robustly average the 3D curve samples that are measurements of the same underlying scene structure. We also apply outlier rejection in this averaging step to discard a huge number of outliers that appear as a result of accidental alignment of image structures with hypothesized 3D curves. As a result, we obtain an optimal depth estimate for all the image curve segments we could reconstruct.
The intermediate results shown above are a cloud of 3D curves. We note that both topologically and geometrically, this representation is still not suitable for use in a wide variety of vision tasks from surface reconstruction to 3D object recognition, due to lack of organization in the 3D space, redundant reconstructions, localization errors and an overall lack of saliency. Below are some examples from Amsterdam House Dataset demonstrating these issues in detail:
In order to merge short, broken and/or duplicate curves and organize this 3D model into a proper 3D graph, we first form "correspondence links" between the 3D curves using shared supporting edge structures in multiple images. Below you can see both an illustration of the process and the visualizations of some real correspondence links, where they are color coded based on how many distinct views are supporting each link:
These links serve as springs, pulling related segments of 3D curves closer in a process that is equivalent to bundle adjustment. We iteratively average bundles of overlapping curve segments to correct for localization errors, since if two curves share a lot of such image edges in multiple views, then they're likely to be the reconstructions of the same scene structure. The iterative averaging converges when the curve motion goes below a certain threshold. The results for this step are shown below:
Finally we organize these results into a cleaner 3D graph representation. We start with a long curve and formulate a pairwise curve merging process. At any given time, we have the 3D graph that was built up until that point along with a query curve that has some overlap with this graph and needs to be merged. The topologies that arise from merging a single curve to a 3D graph are numerous, some of which are illustrated below:
Here, blue curves in the top row are the branches of the existing curve graph, yellow curve is the query curve trying to be merged. Bottom row shows the desired results, with colors indicating different curves.
Our graph organization algorithm scans the query curve samples one by one to merge the ones that have correspondence directly into the graph by finding the closest two samples on the relevant branch and then inserting the query sample between them. The samples that don't have correspondence are used to build up new branches that are added to the graph as soon as their endpoints are determined.
The entire graph organization pipeline, including the iterative averaging step, can be illustrated as follows:
And our previous examples, when fully organized into a graph this way, look like this:
Below, you can view some of results, comparing the 3D drawing to Fabbri and Kimia's Curve Sketch  as well as Furukawa and Ponce's PMVS .
Amsterdam House - Curve Sketch and 3D Drawing, Before & After Graph Organization
Amsterdam House - PMVS
Capitol - 3D Drawing and PMVS
Curvy Vase - 3D Drawing, Before & After Graph Organization
Curvy Vase - PMVS
: "A Multi-Stage Approach to Curve Extraction", Y.Guo, N.Kumar, M.Narayanan and B.Kimia, ECCV 2014
: "3D curve sketch: Flexible curve-based stereo reconstruction and calibration", R. Fabbri and B. Kimia, CVPR 2010
: "Accurate, Dense, and Robust Multiview Stereopsis", Y. Furukawa and J. Ponce, PAMI 2009