Stereo reconstruction from multiperspective panoramas
First Claim
1. A computer-implemented process for computing a 3D reconstruction of a scene, comprising using a computer to perform the following process actions:
- constructing at least two multiperspective panoramas from a collection of single-perspective images of the scene each of which was captured from a different, but co-axial, viewpoint, wherein each multiperspective panorama comprises a plurality of side-by-side columnar images depicting consecutive portions of the scene each of which was obtained from a different one of the single-perspective images; and
computing the 3D reconstruction of the scene from the at least two multiperspective panoramas.
2 Assignments
0 Petitions
Accused Products
Abstract
A system and process for computing a 3D reconstruction of a scene using multiperspective panoramas. The reconstruction can be generated using a cylindrical sweeping approach, or under some conditions, traditional stereo matching algorithms. The cylindrical sweeping process involves projecting each pixel of the multiperspective panoramas onto each of a series of cylindrical surfaces of progressively increasing radii. For each pixel location on each cylindrical surface, a fitness metric is computed for all the pixels projected thereon to provide an indication of how closely a prescribed characteristic of the projected pixels matches. Then, for each respective group of corresponding pixel locations of the cylindrical surfaces, it is determined which location has a fitness metric that indicates the prescribed characteristic of the projected pixels matches more closely than the rest. For each of these winning pixel locations, its panoramic coordinates are designated as the position of the portion of the scene depicted by the pixels projected to that location. Additionally, in some cases a sufficiently horizontal epipolar geometry exists between multiperspective panoramas such that traditional stereo matching algorithms can be employed for the reconstruction. A symmetric pair of multiperspectives panoramas produces the horizontal epipolar geometry. In addition, this geometry is obtained if the distance from the center of rotation to the viewpoints used to capture the images employed to construct the panorama is small in comparison to the distance from the center of rotation to the nearest scene point depicted in the images, or if an off-axis angle is kept small.
306 Citations
61 Claims
-
1. A computer-implemented process for computing a 3D reconstruction of a scene, comprising using a computer to perform the following process actions:
-
constructing at least two multiperspective panoramas from a collection of single-perspective images of the scene each of which was captured from a different, but co-axial, viewpoint, wherein each multiperspective panorama comprises a plurality of side-by-side columnar images depicting consecutive portions of the scene each of which was obtained from a different one of the single-perspective images; and
computing the 3D reconstruction of the scene from the at least two multiperspective panoramas. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)
projecting each pixel of each multiperspective panorama being used to compute the 3D reconstruction onto each of a series of cylindrical surfaces of progressively increasing radii, all of which exceed the radius of the outermost multiperspective panorama employed;
for each pixel location on each cylindrical surface, computing a fitness metric for all the pixels projected from each multiperspective panorama onto the pixel location, wherein said fitness metric provides an indication as to how closely a prescribed characteristic of said projected pixels match each other;
for each respective group of corresponding pixel locations of the cylindrical surfaces, determining which particular location of the group has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest (hereinafter referred to as the winning pixel location), wherein a group of corresponding pixel locations of the cylindrical surfaces are those which depict the same portion of the scene; and
for each winning pixel location, identifying its panoramic coordinates and designating these coordinates to be the position of the portion of the scene depicted by the pixels projected from the multiperspective panoramas to that location.
-
-
7. The process of claim 6, wherein the process action of determining the winning pixel location comprises an act of employing a correlation-based stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
8. The process of claim 6, wherein the process action of determining the winning pixel location comprises an act of employing a global optimization stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
9. The process of claim 6, wherein the process action of computing a 3D reconstruction of the scene further comprises the act of spatially aggregating the computed fitness metric at each pixel location on each cylindrical surface, prior to performing the process action of determining the winning pixel location for each respective group of corresponding pixel locations of the cylindrical surfaces.
-
10. The process of claim 1, wherein the process action of constructing at least two multiperspective panoramas comprises an act of constructing a pair of symmetric multiperspective panoramas, wherein two multiperspective panoramas are symmetric whenever,
the columnar images used to construct each panorama are all obtained from single perspective images whose viewpoints are at the same radius from a common center of rotation, the columnar images used to construct the first panorama of the pair are all centered about a first line forming an off-axis angle in the clockwise direction with respect to a swing line, said swing line being defined by the common center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and the columnar images used to construct the second panorama of the pair are all centered about a second line forming the same off-axis angle with respect to the swing line but in the counterclockwise direction. -
11. The process of claim 10, wherein the process action of computing a 3D reconstruction of the scene comprises the acts of:
-
employing the pair of symmetric multiperspective panoramas;
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
12. The process of claim 1, wherein the process action of constructing at least two multiperspective panoramas comprises an act of constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same radius from a center of rotation, and wherein said radius is less than approximately 0.7 the distance from the center of rotation and the nearest point in the scene depicted in any of the columnar images.
-
13. The process of claim 12, wherein the process action of computing a 3D reconstruction of the scene comprises the acts of:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
14. The process of claim 13, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
15. The process of claim 1, wherein the process action of constructing at least two multiperspective panoramas comprises an act of constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same fixed radius from a center of rotation and which are all laterally centered about a line forming an off-axis angle with respect to a swing line, said swing line being defined by the center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and wherein the off-axis angle does not exceed approximately 15 degrees.
-
16. The process of claim 15, wherein the process action of computing a 3D reconstruction of the scene comprises the acts of:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
17. The process of claim 16, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
18. The process of claim 1, further comprising the act of creating a depth map of the scene from the computed 3D reconstruction.
-
19. A system for computing a 3D reconstruction of a scene, comprising:
-
a general purpose computing device;
a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to, construct at least two multiperspective panoramas from a collection of single-perspective images of the scene each of which was captured from a different, but co-axial, viewpoint, wherein each multiperspective panorama comprises a plurality of side-by-side columnar images depicting consecutive portions of the scene each of which was obtained from a different one of the single-perspective images, and compute the 3D reconstruction of the scene from the at least two multiperspective panoramas. - View Dependent Claims (20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43)
projecting each pixel of each multiperspective panorama being used to compute the 3D reconstruction onto each of a series of cylindrical surfaces of progressively increasing radii, all of which exceed the radius of the outermost multiperspective panorama employed;
for each pixel location on each cylindrical surface, computing a fitness metric for all the pixels projected from each multiperspective panorama onto the pixel location, wherein said fitness metric provides an indication as to how closely a prescribed characteristic of said projected pixels match each other;
for each respective group of corresponding pixel locations of the cylindrical surfaces, determining which particular location of the group has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest (hereinafter referred to as the winning pixel location), wherein a group of corresponding pixel locations of the cylindrical surfaces are those which depict the same portion of the scene; and
for each winning pixel location, identifying its panoramic coordinates and designating these coordinates to be the position of the portion of the scene depicted by the pixels projected from the multiperspective panoramas to that location.
-
-
32. The system of claim 31, wherein the sub-module for determining the winning pixel location comprises a sub-module for employing a correlation-based stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
33. The system of claim 31, wherein the sub-module for determining the winning pixel location comprises a sub-module for employing a global optimization stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
34. The system of claim 31, wherein the program module for computing a 3D reconstruction of the scene further comprises a sub-module for spatially aggregating the computed fitness metric at each pixel location on each cylindrical surface, prior to performing the process action of determining the winning pixel location for each respective group of corresponding pixel locations of the cylindrical surfaces.
-
35. The system of claim 19, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing a pair of symmetric multiperspective panoramas, wherein two multiperspective panoramas are symmetric whenever,
the columnar images used to construct each panorama are all obtained from single perspective images whose viewpoints are at the same radius from a common center of rotation, the columnar images used to construct the first panorama of the pair are all centered about a first line forming an off-axis angle in the clockwise direction with respect to a swing line, said swing line being defined by the common center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and the columnar images used to construct the second panorama of the pair are all centered about a second line forming the same off-axis angle with respect to the swing line but in the counterclockwise direction. -
36. The system of claim 35, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
employing the pair of symmetric multiperspective panoramas;
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
37. The system of claim 19, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same radius from a center of rotation, and wherein said radius is less than approximately 0.7 of the distance from the center of rotation and the nearest point in the scene depicted in any of the columnar images.
-
38. The system of claim 37, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
39. The system of claim 38, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
40. The system of claim 19, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same fixed radius from a center of rotation and which are all laterally centered about a line forming an off-axis angle with respect to a swing line, said swing line being defined by the center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and wherein the off-axis angle does not exceed approximately 15 degrees.
-
41. The system of claim 40, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
42. The system of claim 41, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
43. The system of claim 19, further comprising a program module for creating a depth map of the scene from the computed 3D reconstruction.
-
44. A computer-readable memory for computing a 3D reconstruction of a scene, comprising:
-
a computer-readable storage medium; and
a computer program comprising program modules stored in the storage medium, wherein the storage medium is so configured by the computer program that it causes a computer to, construct at least two multiperspective panoramas from a collection of single-perspective images of the scene each of which was captured from a different, but co-axial, viewpoint, wherein each multiperspective panorama comprises a plurality of side-by-side columnar images depicting consecutive portions of the scene each of which was obtained from a different one of the single-perspective images, and compute the 3D reconstruction of the scene from the at least two multiperspective panoramas. - View Dependent Claims (45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61)
projecting each pixel of each multiperspective panorama being used to compute the 3D reconstruction onto each of a series of cylindrical surfaces of progressively increasing radii, all of which exceed the radius of the outermost multiperspective panorama employed;
for each pixel location on each cylindrical surface, computing a fitness metric for all the pixels projected from each multiperspective panorama onto the pixel location, wherein said fitness metric provides an indication as to how closely a prescribed characteristic of said projected pixels match each other;
for each respective group of corresponding pixel locations of the cylindrical surfaces, determining which particular location of the group has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest (hereinafter referred to as the winning pixel location), wherein a group of corresponding pixel locations of the cylindrical surfaces are those which depict the same portion of the scene; and
for each winning pixel location, identifying its panoramic coordinates and designating these coordinates to be the position of the portion of the scene depicted by the pixels projected from the multiperspective panoramas to that location.
-
-
50. The computer-readable memory of claim 49, wherein the sub-module for determining the winning pixel location comprises a sub-module for employing a correlation-based stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
51. The computer-readable memory of claim 49, wherein the sub-module for determining the winning pixel location comprises a sub-module for employing a global optimization stereo depth technique to identify which particular location of each group of corresponding pixel locations of the cylindrical surfaces has a computed fitness metric that indicates the prescribed characteristic of said projected pixels matches more closely than the rest.
-
52. The computer-readable memory of claim 49, wherein the program module for computing a 3D reconstruction of the scene further comprises a sub-module for spatially aggregating the computed fitness metric at each pixel location on each cylindrical surface, prior to performing the process action of determining the winning pixel location for each respective group of corresponding pixel locations of the cylindrical surfaces.
-
53. The computer-readable memory of claim 44, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing a pair of symmetric multiperspective panoramas, wherein two multiperspective panoramas are symmetric whenever,
the columnar images used to construct each panorama are all obtained from single perspective images whose viewpoints are at the same radius from a common center of rotation, the columnar images used to construct the first panorama of the pair are all centered about a first line forming an off-axis angle in the clockwise direction with respect to a swing line, said swing line being defined by the common center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and the columnar images used to construct the second panorama of the pair are all centered about a second line forming the same off-axis angle with respect to the swing line but in the counterclockwise direction. -
54. The computer-readable memory of claim 53, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
employing the pair of symmetric multiperspective panoramas;
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
55. The computer-readable memory of claim 44, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same radius from a center of rotation, and wherein said radius is less than approximately 0.7 of the distance from the center of rotation and the nearest point in the scene depicted in any of the columnar images.
-
56. The computer-readable memory of claim 55, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
57. The computer-readable memory of claim 56, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
58. The computer-readable memory of claim 44, wherein the program module for constructing at least two multiperspective panoramas comprises a sub-module for constructing each multiperspective panorama using columnar images obtained from single perspective images whose viewpoints are all at the same fixed radius from a center of rotation and which are all laterally centered about a line forming an off-axis angle with respect to a swing line, said swing line being defined by the center of rotation and the viewpoint associated with each single-perspective images from which the respective columnar images were obtained, and wherein the off-axis angle does not exceed approximately 15 degrees.
-
59. The computer-readable memory of claim 58, wherein the program module for computing a 3D reconstruction of the scene comprises sub-modules for:
-
compensating for any global vertical scale difference in each multiperspective panorama; and
using any traditional stereo matching algorithm requiring a horizontal epipolar geometry to designate the coordinates of the portion of the scene depicted by each corresponding pixel location of the multiperspective panoramas.
-
-
60. The computer-readable memory of claim 59, wherein there are more than two multiperspective panoramas used to compute the 3D reconstruction of the scene, and wherein the traditional stereo matching algorithm is a multi-image stereo matching process.
-
61. The computer-readable memory of claim 44, further comprising a program module for creating a depth map of the scene from the computed 3D reconstruction.
Specification