Enhanced object reconstruction

• US 9,234,749 B2
• Filed: 06/06/2012
• Issued: 01/12/2016
• Est. Priority Date: 01/07/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gu, Jin
• Primary Examiner(s): Carter, Aaron W
• Application Number:
  US13/489,588
• Publication Number:
  US 20120301013A1
• Time in Patent Office: 1,315 Days
• Field of Search: 382/154, 382/285, 345419-427, 356 12- 14, 348 42- 60
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Processing images includes projecting an infra-red pattern onto a three-dimensional object and producing a first image, a second image, and a third image of the three-dimensional object while the pattern is projected on the three-dimensional object. The first image and the second image include the three-dimensional object and the pattern. The first image and the second image are produced by capturing at a first camera and a second camera, respectively, light filtered through an infra-red filter. The third image includes the three-dimensional object but not the pattern. Processing the images also includes establishing a first-pair correspondence between a portion of pixels in the first image and a portion of pixels in the second image. Processing the images further includes constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.

Claims

1. A method for determining three-dimensional information, the method comprising:

identifying a single reference element created by a fixed pattern projected onto a three-dimensional object in each of two images, the fixed pattern including a background region, an element, and the single reference element, wherein the element and the single reference element are distinct from the background region, and wherein the single reference element is distinct from the element;

determining a correspondence between portions of the two images based on the single reference element created by the fixed pattern;

determining three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images; and

generating a three dimensional surface model using the three-dimensional information based on projection of only the fixed pattern onto the three-dimensional object.

2. The method for determining three-dimensional information of claim 1, further comprising:

projecting the fixed pattern onto the three-dimensional object.

3. The method for determining three-dimensional information of claim 2, wherein the fixed pattern comprises a stripe.

4. The method for determining three-dimensional information of claim 3, wherein the single reference element comprises a wide stripe, wherein the wide stripe is wider than the stripe.

5. The method for determining three-dimensional information of claim 1, wherein the fixed pattern further comprises a plurality of repeating elements.

6. The method for determining three-dimensional information of claim 1, wherein determining the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images comprises:

determining a three-dimensional point cloud about the three-dimensional object based on the correspondence between the portions of the two images.

7. The method for determining three-dimensional information of claim 6, wherein determining the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images further comprises:

determining a three-dimensional surface model of the three-dimensional object.

8. The method for determining three-dimensional information of claim 1, wherein the three-dimensional object comprises a human face.

9. The method for determining three-dimensional information of claim 1, wherein the pattern projected onto a three-dimensional object results in one or more elements having a depth discontinuity.

10. The method for determining three-dimensional information of claim 9, wherein determining the correspondence between portions of the two images based on the single reference elements comprises:

performing a disparity propagation routine that identifies depth discontinuities.

28. The method for determining three-dimensional information of claim 1, wherein the element and the single reference element of the fixed pattern comprise light projected onto the object, and wherein the background region of the fixed pattern comprises an area separating the projected light.

11. A system configured to determine three-dimensional information, the system comprising:

a computer processor; and

a memory communicatively coupled with and readable by the computer processor and having stored therein processor-readable instructions which, when executed by the computer processor, cause the computer processor to;

identify a single reference element created by a fixed pattern projected onto a three-dimensional object in each of two images, the fixed pattern including a background region, an element, and the single reference element, wherein the element and the single reference element are distinct from the background region, and wherein the single reference element is distinct from the element;

determine a correspondence between portions of the two images based on the single reference element created by the fixed pattern;

determine three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images; and

generate a three dimensional surface model using the three-dimensional information based on projection of only the fixed pattern onto the three-dimensional object.

12. The system for determining three-dimensional information of claim 11, further comprising:

a projector configured to project the fixed pattern onto the three-dimensional object.

13. The system for determining three-dimensional information of claim 12, wherein the projector is configured to project the fixed pattern in infrared.

14. The system for determining three-dimensional information of claim 12, wherein the fixed pattern projected by the projector comprises a stripe.

15. The system for determining three-dimensional information of claim 11, further comprising:

a first camera configured to capture a first image of the two images; and

a second camera configured to capture a second image of the two images.

16. The system for determining three-dimensional information of claim 15, wherein the first camera or the second camera comprises an infrared filter.

17. The system for determining three-dimensional information of claim 11, wherein the instructions that cause the computer processor to determine the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images comprises processor-readable instructions that cause the computer processor to:

determine a three-dimensional point cloud about the three-dimensional object based on the correspondence between the portions of the two images.

18. The system for determining three-dimensional information of claim 11, wherein the instructions that cause the computer processor to determine the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images comprises processor-readable instructions that cause the computer processor to:

determine a three-dimensional surface model of the three-dimensional object.

19. The system for determining three-dimensional information of claim 11, wherein the pattern projected onto a three-dimensional object results in one or more elements having a depth discontinuity.

20. The system for determining three-dimensional information of claim 19, wherein the instructions that cause the computer processor to determine the correspondence between portions of the two images based on the single reference elements comprises processor-readable instructions that cause the computer processor to:

perform a disparity propagation routine that identifies depth discontinuities of the pattern projected onto the three-dimensional object.

21. An apparatus for determining three-dimensional information, the apparatus comprising:

means for identifying a single reference element created by a fixed pattern projected onto a three-dimensional object in each of two images, the fixed pattern including a background region, an element, and the single reference element, wherein the element and the single reference element are distinct from the background region, and wherein the single reference element is distinct from the element;

means for determining a correspondence between portions of the two images based on the single reference element created by the fixed pattern;

means for determining three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images; and

means for generating a three dimensional surface model using the three-dimensional information based on projection of only the fixed pattern onto the three-dimensional object.

22. The apparatus for determining three-dimensional information of claim 21, further comprising:

means for projecting the fixed pattern in infrared onto the three-dimensional object.

23. The apparatus for determining three-dimensional information of claim 21, wherein the means for determining the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images comprises:

means for determining a three-dimensional point cloud about the three-dimensional object based on the correspondence between the portions of the two images.

24. A non-transitory processor-readable medium for determining three-dimensional information, comprising processor-readable instructions configured to cause one or more processors to:

identify a single reference element created by a fixed pattern projected onto a three-dimensional object in each of two images, the fixed pattern including a background region, an element that is distinct from the background region, and the single reference element that is distinct from the background region and the element;

determine a correspondence between portions of the two images based on the single reference element created by the fixed pattern;

determine three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images; and

generate a three dimensional surface model using the three-dimensional information based on projection of only the fixed pattern onto the three-dimensional object.

25. The non-transitory processor-readable medium for determining three-dimensional information of claim 24, wherein the instructions that cause the computer processor to determine the three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images comprises processor-readable instructions that cause the computer processor to:

determine a three-dimensional point cloud about the three-dimensional object based on the correspondence between the portions of the two images.

26. A method for determining three-dimensional information, the method comprising:

identifying a non-repeating reference element created by a fixed pattern projected onto a three-dimensional object at a single position in each of two images, the fixed pattern being fixed relative to a light source that is projecting the fixed pattern, the fixed pattern including a first portion having a first repetitive component created by light projected from the light source and a second portion having a second repetitive component created by an absence of light projected from the light source, wherein the non-repeating reference element is distinct from the first portion and the second portion;

determining a correspondence between portions of the two images based on the non-repeating reference element created by the fixed pattern;

determining three-dimensional information about the three-dimensional object based on the correspondence between the portions of the two images; and

generating a three dimensional surface model using the three-dimensional information based on projection of only the fixed pattern onto the three-dimensional object at the single position.

27. The method for determining three-dimensional information of claim 26, wherein the non-repeating reference element comprises light projected from the light source.