Producing 3D images from captured 2D video

• US 9,300,947 B2
• Filed: 03/24/2011
• Issued: 03/29/2016
• Est. Priority Date: 03/24/2011
• Status: Active Grant

• Current Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Original Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Inventors: Park, Minwoo, Luo, Jiebo, Gallagher, Andrew Charles
• Primary Examiner(s): Vaughn, Jr., William C
• Assistant Examiner(s): PEREZ FUENTES, LUIS M
• Application Number:
  US13/070,670
• Publication Number:
  US 20120242794A1
• Time in Patent Office: 1,832 Days
• Field of Search: 348/46, 382/154
• US Class Current: 1/1
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/211 using temporal multiplexing
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/264 using the relative movement of objects in two video frames or fields

Abstract

A method of producing a stereo image from a temporal sequence of digital images, comprising: receiving a temporal sequence of digital images; analyzing pairs of digital images to produce corresponding stereo suitability scores, wherein the stereo suitability score for a particular pair of images is determined responsive to the relative positions of corresponding features in the particular pair of digital image; selecting a pair of digital images including a first image and a second image based on the stereo suitability scores; using a processor to analyze the selected pair of digital images to produce a motion consistency map indicating regions of consistent motion, the motion consistency map having an array of pixels; producing a stereo image pair including a left view image and a right view image by combining the first image and the second image responsive to the motion consistency map; and storing the stereo image pair in a processor-accessible memory.

Claims

1. A method of producing a stereo image from a temporal sequence of digital images, comprising:

receiving a temporal sequence of digital images captured by an image capture device;

using a processor to analyze pairs of digital images from the temporal sequence of digital images to produce corresponding stereo suitability scores, wherein the stereo suitability score for a particular pair of images is based on a set of motion vectors between corresponding points in the pair of digital images and epipolar geometry characteristics determined from the set of motion vectors, and wherein the suitability score is based on statistics of inliers and outliers of epipolar flow associated with the epipolar geometry characteristics;

selecting a pair of digital images including a first image and a second image based on the stereo suitability scores;

using a processor to analyze the selected pair of digital images to produce a motion consistency map indicating regions of consistent motion, the motion consistency map having an array of pixels;

using a processor to analyze the motion consistency map to determine whether the selected pair of digital images requires adjustment;

if the selected pair of digital images requires adjustment, producing a stereo image pair including a left view image and a right view image by combining the first image and the second image responsive to the motion consistency map; and

storing the stereo image pair in a processor-accessible memory.

2. The method of claim 1 wherein the motion consistency map is produced by:

identifying at least one rightward motion image region in the pair of digital images where the positions of a set of features in the image region in the first image are to the right of the positions of a corresponding set of features in the second image; and

assigning a first pixel value to pixels in the motion consistency map corresponding to the at least one rightward motion image region, and assigning a second pixel value to the remaining pixels in the motion consistency map.

3. The method of claim 1 wherein the motion consistency map is produced by:

identifying at least one leftward motion image region in the pair of digital images where the positions of a set of features in the image region in the first image are to the right of the positions of a corresponding set of features in the second image; and

assigning a first pixel value to pixels in the motion consistency map corresponding to the at least one leftward motion image region, and assigning a second pixel value to the remaining pixels in the motion consistency map.

4. The method of claim 1 wherein the motion consistency map is produced by:

identifying at least one rightward motion image region in the pair of digital images where the positions of a set of features in the image region in the first image are to the left of the positions of a corresponding set of features in the second image and assigning a first pixel value to pixels in the motion consistency map corresponding to the at least one rightward motion image region;

identifying at least one leftward motion image region in the pair of digital images where the positions of a set of features in the image region in the first image are to the right of the positions of a corresponding set of features in the second image and assigning a second pixel value to pixels in the motion consistency map corresponding to the at least one leftward motion image region; and

assigning either the first pixel value or the second pixel value to the remaining pixels in the motion consistency map.

5. The method of claim 1 wherein the motion consistency map is produced by first producing a binary motion consistency map, each pixel of the binary motion consistency map having either a first pixel value or a second pixel value, and then forming the motion consistency map by applying a smoothing operator to the binary motion consistency map to the blur a boundary between a first pixel value region and a second pixel value region to provide a smooth transition between the two regions.

6. The method of claim 1 wherein the motion consistency map specifies a weighting value for blending the first and second images, and wherein pixel values for the left view image and the right view image are produced by combining corresponding pixel values from the first and second images according to the corresponding weighting values in the motion consistency map.

7. The method of claim 6 wherein the weighting values provide an indication of the relative fraction of the first and second images that should be combined to produce the left view image, and wherein complementary fractions are used to combine the first and second images to produce the right view image.

8. The method of claim 1 wherein the temporal sequence of digital images is a digital video.

9. The method of claim 1 wherein the temporal sequence of digital images is a set of images captured in a burst capture mode.

10. The method of claim 1 wherein the process of analyzing a pair of digital images to produce a corresponding stereo suitability scores includes:

determining the suitability score training a support vector machine classifier responsive to the set of motion vectors and the epipolar geometry characteristics, wherein the support vector machine classifier is trained using a training set containing some pairs of digital images that are known to be suitable for producing stereo images and other pairs of digital images that are known to be unsuitable for producing stereo images.

11. The method of claim 10 wherein the set of motion vectors are determined by tracking a set of interest points between the pair of digital images.

12. The method of claim 11 wherein the set of interest points are tracked using a Kanade-Lucas-Tomasi feature tracking algorithm.

13. The method of claim 10 wherein the set of features further includes one or more features determined by applying statistical calculations to the set of motion vectors.

14. The method of claim 10 wherein the classifier is a support vector machine classifier.

15. The method of claim 1 further including displaying the stereo image pair using a stereo image display system.

16. The method of claim 15 wherein the stereo image display system is an anaglyph display system including a color image forming device, and wherein the left view images are combined to form an anaglyph image, which when displayed using the color image forming device is adapted to be viewed by an observer wearing anaglyph glasses having different color filters for the observer'"'"'s left and right eyes.

17. The method of claim 16 wherein the color image forming device is a color softcopy display or a color hardcopy printer.

18. The method of claim 15 wherein the stereo image display system includes a lenticular stereoscopic display, a barrier stereoscopic display or a polarizing stereoscopic display.

19. The method of claim 15 wherein the stereo image display system includes:

a softcopy display for displaying the left view image and the right view image in an alternating sequence;

and shutter glasses synchronized with the softcopy display such that when the softcopy display is displaying the left view image the shutter glasses block light to an observer'"'"'s right eye, and when the softcopy display is displaying the right view image the shutter glasses block light to the observer'"'"'s left eye.