Creating 3D images of objects by illuminating with infrared patterns

• US 20060210146A1
• Filed: 01/09/2006
• Published: 09/21/2006
• Est. Priority Date: 01/07/2005
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: Gu, Jin
• Granted Patent: US 7,430,312 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 2201/121 using special illumination
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

According to a general aspect, processing images includes projecting an infra-red pattern onto a three-dimensional object and producing a first image, a second image, and a third image of the three-dimensional object while the pattern is projected on the three-dimensional object. The first image and the second image include the three-dimensional object and the pattern. The first image and the second image are produced by capturing at a first camera and a second camera, respectively, light filtered through an infra-red filter. The third image includes the three-dimensional object but not the pattern. Processing the images also includes establishing a first-pair correspondence between a portion of pixels in the first image and a portion of pixels in the second image. Processing the images further includes constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.

Claims

1. A method comprising:

projecting an infra-red pattern onto a three-dimensional object;

producing a first image of the three-dimensional object while the pattern is projected on the three-dimensional object, the first image (i) including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a first camera light filtered through an infra-red filter;

producing a second image of the three-dimensional object while the pattern is projected on the three-dimensional object, the second image (i) including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a second camera light filtered through an infra-red filter, and the first and second cameras arranged as a first stereo pair having a known physical relationship;

establishing a first-pair correspondence between a portion of the pixels of the first image and a portion of the pixels of the second image;

producing a third image of the three-dimensional object while the pattern is projected on the three-dimensional object, the third image (i) including the three-dimensional object but not the pattern and (ii) being a two-dimensional digital image including pixels; and

constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.

2. The method of claim 1 wherein projecting the infra-red pattern comprises projecting a non-random infra-red pattern.

3. The method of claim 1 wherein establishing the first-pair correspondence comprises:

determining a correspondence between an initial pixel in the first image and a corresponding pixel in the second image; and

determining a correspondence between additional pixels in the first image and corresponding pixels in the second image, based on the correspondence between the initial pixel in the first image and its corresponding pixel in the second image.

4. The method of claim 1 wherein establishing the first-pair correspondence comprises:

determining a correspondence between a first initial pixel located on a first particular horizontal line in the first image and a first corresponding pixel that corresponds to the first initial pixel, wherein the first corresponding pixel is located on the first particular horizontal line in the second image;

determining a correspondence between additional pixels located on the first particular horizontal line in the first image and corresponding pixels that correspond to the additional pixels, wherein the corresponding pixels are located on the first particular horizontal line in the second image;

determining a correspondence between a second initial pixel located on a second particular horizontal line in the first image and a second corresponding pixel that corresponds to the second initial pixel, wherein the second corresponding pixel is located on the second particular horizontal line in the second image; and

determining a correspondence between additional pixels located on the second particular horizontal line in the first image and corresponding pixels that correspond to the additional pixels, wherein the corresponding pixels are located on the second particular horizontal line in the second image.

5. The method of claim 1 wherein establishing the first-pair correspondence comprises:

determining a correspondence between an initial pixel in each horizontal line in the first image and a corresponding pixel in each horizontal line in the second image; and

determining a correspondence between additional pixels in the first image and corresponding pixels in the second image, based on the correspondence between the initial pixel in each horizontal line in the first image and its corresponding pixel in each horizontal line in the second image.

6. The method of claim 1 wherein the pattern comprises vertical stripes.

7. The method of claim 5 wherein the first initial pixel is a centroid pattern pixel calculated from the pattern pixels in the first particular horizontal line.

8. The method of claim 4, wherein determing the correspondence for at least one of the additional pixels located on the second particular horizontal line in the first image is based on the correspondence determined for at least one other pixel located in the second particular horizontal line.

9. The method of claim 4, wherein determing the correspondence for at least one of the additional pixels located on the second particular horizontal line in the first image is based on the correspondence determined for at least one pixel located in the first particular horizontal line.

10. The method of claim 9 wherein the at least one pixel located in the first particular horizontal line is in a common stripe edge with the at least one of the additional pixels located in the second particular horizontal line.

11. The method of claim 1 wherein constructing the two-dimensional image that depicts the three-dimensional construction comprises:

forming a first set of three-dimensional points based on the first-pair correspondence; and

producing a first three-dimensional surface model based on the first set of three-dimensional points.

12. The method of claim 1 further comprising:

producing a fourth image of the three-dimensional object while the pattern is projected on the three-dimensional object, the fourth image being a two-dimensional digital image including pixels and being produced by capturing at a fourth camera light filtered through an infra-red filter;

producing a fifth image of the three-dimensional object while the pattern is projected on the three-dimensional object, the fifth image being a two-dimensional digital image including pixels and being produced by capturing at a fifth camera light filtered through an infra-red filter, the fourth and fifth cameras arranged as a second stereo pair having a known physical relationship; and

establishing a second-pair correspondence between a portion of the pixels of the fourth image and a portion of the pixels of the fifth image,

wherein constructing the two-dimensional image that depicts the three-dimensional construction of the three-dimensional object is further based on the second-pair correspondence.

13. The method of claim 12 wherein constructing the two-dimensional image that depicts the three-dimensional construction comprises:

producing a first three-dimensional surface model based on the first-pair correspondence;

producing a second three-dimensional surface model based on the second-pair correspondence; and

registering the first and second three-dimensional surface models.

14. The method of claim 13 wherein registering comprises:

determining a common surface in the first and the second three-dimensional surface models;

producing an initial estimate for a registration matrix based on the common surface; and

determining the closest points between the first and the second three-dimensional surface models based on the initial estimate for a registration matrix.

15. The method of claim 13 wherein:

producing the first three-dimensional surface model comprises;

forming a first set of three-dimensional points based on the first-pair correspondence; and

producing the first three-dimensional surface model based on the first set of three-dimensional points, and

producing the first three-dimensional surface model comprises;

forming a second set of three-dimensional points based on the second-pair correspondence; and

producing the second three-dimensional surface model based on the second set of three-dimensional points.

16. The method of claim 13 further comprising:

integrating, after the registering, the first and second three-dimensional surface models to produce an integrated three-dimensional surface model; and

providing texture to the integrated three-dimensional surface model.

17. The method of claim 1 wherein producing the third image comprises producing the third image by capturing non-filtered light at a third camera.

18. The method of claim 17 wherein the third camera is a texture camera.

19. A system comprising:

a first stereo camera pair including a first camera coupled to a second camera;

a second stereo camera pair including a third camera coupled to a fourth camera;

a set of four infra-red filters, with a separate one of the four infra-red filters operatively coupled to each of four cameras;

a projector; and

a computer readable medium coupled to each of the four cameras and to the projector, and including instructions for performing at least the following;

projecting an infra-red pattern from the projector onto a three-dimensional object;

producing a first image of the three-dimensional object while the pattern is projected on the three-dimensional object, the first image (i) including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a first camera light filtered through an infra-red filter;

producing a second image of the three-dimensional object while the pattern is projected on the three-dimensional object, the second image (i) including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a second camera light filtered through an infra-red filter, and the first and second cameras arranged as a first stereo pair having a known physical relationship;

establishing a first-pair correspondence between a portion of the pixels of the first image and a portion of the pixels of the second image;

producing a third image of the three-dimensional object while the pattern is projected on the three-dimensional object, the second third image (i) including the three-dimensional object but not the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing light at a texture camera; and

constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.

20. The system of claim 19 wherein the projector includes a lighting source capable of producing light in the visible spectrum and in the infrared spectrum.

21. The system of claim 20 wherein the projector includes a fifth infra-red filter.

22. The system of claim 19 wherein the computer readable medium comprises one or more of a processing device and a storage device.

23. A computer readable medium including instructions for performing at least the following:

accessing a first image captured of a three-dimensional object, the first image having been captured while a pattern is projected on the three-dimensional object, the first image (i). including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a first camera light filtered through an infra-red filter;

accessing a second image captured of the three-dimensional object, the second image having been captured while the pattern is projected on the three-dimensional object, the second image (i) including the three-dimensional object and the pattern, (ii) being a two-dimensional digital image including pixels, and (iii) being produced by capturing at a second camera light filtered through an infra-red filter;

establishing a first-pair correspondence between a portion of the pixels of the first image and a portion of the pixels of the second image, the first-pair correspondence being established based on the first and second cameras having been arranged as a first stereo pair having a known physical relationship while the first and second images were captured;

accessing a third image captured of the three-dimensional object, the third image having been captured while the pattern is projected on the three-dimensional object, the third image (i) including the three-dimensional object but not the pattern, (ii) being a two-dimensional digital image including pixels; and

constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.