Apparatus and methods for determining the three-dimensional shape of an object using active illumination and relative blurring in two-images due to defocus

• US 6,229,913 B1
• Filed: 07/18/1997
• Issued: 05/08/2001
• Est. Priority Date: 06/07/1995
• Status: Expired due to Term

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Nayar, Shree, Wantanabe, Masahiro, Noguchi, Minori
• Primary Examiner(s): Bella, Matthew C.
• Application Number:
  US08/776,799
• Time in Patent Office: 1,390 Days
• Field of Search: 382/154, 382/212, 382/279, 382/285, 250/201.4, 250/201.6, 355/55
• US Class Current: 382/154
• CPC Class Codes: G01C 3/32 by focusing the object, e.g. on a ground glass screen
• CPC Class Codes: G02B 30/52 the 3D volume being constructed from a stack or sequence of 2D planes, e.g. depth sampling systems
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/66 Trinkets, e.g. shirt buttons or jewellery items recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 13/236 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method and apparatus for mapping depth of an object (22) in a preferred arrangement uses a projected light pattern to provide a selected texture to the object (22) along the optical axis (24) of observation. An imaging system senses (32, 34) first and second images of the object (22) with the projected light pattern and compares the defocused of the projected pattern in the images to determine relative depth of elemental portions of the object (22).

Claims

1. A method for mapping a three-dimensional stricture by depth from defocus, comprising the steps of:

(a) illuminating said structure with a preselected illumination pattern;

(b) sensing at least a first image and a second image of said illuminated structure with different imaging parameters; and

(c) determining a relative blur between at least one elemental portion of said first image and at least one elemental portion of said second image which correspond to the same portion of said three-dimensional structure to identify depth of corresponding elemental portions of said three-dimensional structure.

2. The method of claim 1 wherein said illumination comprises illumination with a two dimensional illumination pattern.

3. The method of claim 2 wherein said illumination is with a rectangular grid having selected horizontal and vertical grid spacing of rectangular transparent and opaque elements forming a checkerboard pattern.

4. The method of claim 3 wherein said sensing comprises sensing using an array of sensing elements having horizontal and vertical array spacings that are integral sub-multiples of said horizontal and vertical grid spacing in said sensed images.

5. The method of claim 1 wherein said first image and said second image are formed with a telecentric lens system.

6. A method for mapping a three-dimensional structure by depth from defocus, comprising the steps of:

(a) illuminating said structure with a preselected illumination pattern comprising a rectangular grid projected along an optical axis;

(b) sensing at least a first image and a second image of said illuminated structure from said optical axis using a constant magnification imaging system and at imaging planes with different locations with respect to a focal plane of said imaging system; and

(c) determining a relative blur between at least one elemental portion of said first image and at least one elemental portion of said second image which correspond to the same portion of said three-dimensional structure to identify depth of corresponding elemental portions of said three dimensional structure.

7. The method of claim 6 wherein said images are sensed using first and second sensing arrays of sensing elements arranged in a rectangular pattern with selected element spacing in each direction of said array.

8. The method of claim 7 wherein said rectangular grid has a checkerboard pattern with selected grid periodicity.

9. The method of claim 8 wherein said grid periodicity is selected to provide a grid image on said sensing arrays wherein said grid periodicity is an integral multiple of said corresponding element spacing.

10. The method of claim 9, wherein said grid periodicity is selected to provide a grid image with a period substantially equal to twice said element spacing, and said grid image is aligned with said array in two orthogonal directions.

11. The method of claim 9, wherein said grid image periodicity is substantially equal to four times said pixel width and said grid image is shifted on said array by one eighth of said grid image periodicity.

12. The method of claim 6, wherein said light source is a monochromatic laser light source.

13. The method of claim 9, wherein said sensing step comprises sensing at least two depth images of said scene formed by said laser light and at least one brightness image of said scene formed by ambient light, and said determining step comprises measuring a relative blur between said sensed laser light images.

14. The method of claim 6, wherein said first image is sensed at a position corresponding to a near focused plane and said second image is sensed at a position corresponding to a far focused plane.

15. The method of claim 6, wherein said illumination grid is selected so as to produce an illumination pattern which generates multiple spatial frequencies.

16. The method of claim 6, wherein said illuminating step further comprises using half-mirror optics to reflect said illumination pattern prior to illuminating said scene, and said sensing step further comprises passing said scene images through said half-mirror optics prior to sensing said scene, such that said illumination pattern and said scene images pass along a common optical axis.

17. The method of claim 6, wherein said illuminating step further comprises using polarization optics to polarize said illumination pattern prior to illuminating said scene, and said sensing step further comprises passing said scene images through polarization optics prior to sensing said scene.

18. The method of claim 6, wherein said determining step further comprises:

(i) converting said sensed images into digital signals on a pixel by pixel basis; and

(ii) convolving said digital signals on a pixel by pixel basis to determine power measurement signals that correspond to the fundamental frequency of said illumination pattern at each of said pixels for each sensed scene image.

19. The method of claim 18, wherein said measuring step further comprises:

(iii) correcting said power measurement signals for mis-registration on a pixel by pixel basis, such that any errors introduced into said power measurement signals because of misalignment between said sensing pixels of said array and said illumination pattern is corrected.

20. The method of claim 19, wherein said correcting step comprises taking the sum of the squares of said measurement signal at four neighboring pixels.

21. The method of claim 18, wherein said measuring step further comprises:

(iii) normalizing said power measurement signals on a pixel by pixel basis.

22. The method of claim 18, wherein said measuring step further comprises:

(iii) comparing said power measurement signals for one of said sensed images, on a pixel by pixel basis, with determined power measurements for a second of said sensed images to determine said depth information at each of said pixels.

23. The method of claim 6, wherein said determination step comprises arranging said pixel by pixel depth information as a depth map.

24. The method of claim 23, further comprising the step of displaying said depth map as a wireframe image.

25. The method of claim 13, wherein said determination step comprises arranging said pixel by pixel depth information as a depth map, further comprising the step of constructing a texture mapped three-dimensional display from said sensed brightness image and said depth map.

26. Apparatus for measuring a three-dimensional structure of a scene by depth from defocus, comprising:

(a) active illumination means for illuminating the scene with a preselected illumination pattern;

(b) sensor means, optically coupled to said illuminating means, for sensing at least a first image and a second image of the scene with differing optical or imaging parameters;

(c) depth measurement means, coupled to said sensor means, for measuring a relative blur between at least one elemental portion of said first image and at least one elemental portion of said second image which correspond to the same portion of said three-dimensional structure to identify depth of said three-dimensional structure; and

(d) scene recovery means, coupled to said measurement means, for reconstructing said three-dimensional structure of said sensed scene from said measured relative blur of said sensed images.

27. The apparatus of claim 26, wherein said sensor means comprises a plurality of sensors, each sensor having X*Y pixels of predetermined width to form an X*Y sensing grid, said depth measurement means measuring said relative blur on a pixel by pixel basis over said X*Y pixel grid, such that depth information is obtained for each of said pixels within said X*Y grid.

28. The apparatus of claim 27, wherein said active illumination means comprises:

(i) an illumination base;

(i) a light source coupled to said illumination base; and

(ii) a spectral filter having said preselected illuminating pattern coupled to said illumination base, such that light from said light source passes through said spectral filter to form said preselected illumination pattern.

29. The apparatus of claim 28, wherein said preselected illumination pattern of said spectral filter is optimized so that a small variation in the degree of defocus sensed by said sensor means results in a large variation in the relative blur measured by said depth measurement means.

30. The apparatus of claim 29, wherein said optimized illumination pattern is a rectangular grid pattern.

31. The apparatus of claim 30, wherein said optimized illumination pattern comprises a pattern having a period being substantially equal to twice said pixel width and a phase shift being substantially equal to zero with respect to said sensing grid, in two orthogonal directions.

32. The apparatus of claim 30, wherein said optimized illumination pattern comprises a pattern having a period being substantially equal to four times said pixel width and a phase shift being substantially equal to one eighth of said pixel width with respect to said sensing grid, in two orthogonal directions.

33. The apparatus of claim 28, wherein said light source is a Xenon lamp.

34. The apparatus of claim 28, wherein said light source is a monochromatic laser.

35. The apparatus of claim 34, wherein said sensor means further comprises:

(i) a sensor base;

(i) first and second depth sensors, coupled to said sensor base, for sensing depth images of said scene formed by said laser light, such that said depth measurement means measure a relative blur between said sensed laser light images; and

(ii) at least one brightness sensor, coupled to said sensor base, for sensing an image of said scene formed by ambient light.

36. The apparatus of claim 26, wherein said sensor means comprises:

(i) a sensor base;

(ii) a lens, coupled to said sensor base and optically coupled to said illuminating means, for receiving scene images;

(iii) a beamsplitter, coupled to said sensor base and optically coupled to said lens, for splitting said scene images into two split scene images; and

(iv) first and second sensors, coupled to said sensor base, wherein said first sensor is optically coupled to said beamsplitter such that a first of said split scene images is incident on said first sensor and said second sensor is optically coupled to said beamsplitter such that a second of said split scene images is incident on said second sensor.

37. The apparatus of claim 36, wherein said sensor means further comprises:

(v) an optical member having an aperture, coupled to said sensor base in a position between said lens and said beamsplitter, being optically coupled to both said lens and said beamsplitter such that images received by said lens are passed through said aperture and are directed toward said beamsplitter.

38. The apparatus of claim 36, wherein said first sensor is at a position corresponding to a near focused plane in said sensed scene, and said second sensor is at a position corresponding to a far focused plane in said sensed scene.

39. The apparatus of claim 38, wherein said spectral filter includes an illumination pattern capable of generating multiple spatial frequencies for each image sensed by said first and second sensors.

40. The apparatus of claim 26, further comprising:

(e) a support member, coupled to said active illumination means and said sensor means; and

(f) a half-mirror, coupled to said support member at an optical intersection of said active illumination means and said sensor means, such that said preselected illumination pattern is reflected by said half-mirror prior to illuminating said scene, and such that said scene images pass through said half-mirror prior to being sensed by said sensor means, whereby said illumination pattern and said scene images pass through coaxial optical paths.

41. The apparatus of claim 26, further comprising:

(e) a support member, coupled to said active illumination means and said sensor means; and

(f) a half-mirror, coupled to said support member at an optical intersection of said active illumination means and said sensor means, such that said preselected illumination pattern passes through said half-mirror prior to illuminating said scene, and such that said scene images are reflected by said half-mirror prior to being sensed by said sensor means, whereby said illumination pattern and said scene images pass through coaxial optical paths.

42. The apparatus of claim 26, further comprising:

(e) a support member, coupled to said active illumination means and said sensor means; and

(f) a polarization filter, coupled to said support member at an optical intersection of said active illumination means and said sensor means, such that said preselected illumination pattern is reflected by said polarization filter prior to illuminating said scene, and such that said scene images pass through said polarization filter prior to being sensed by said sensor means, whereby said illumination pattern incident on said scene and said sensed scene images are both polarized in controlled polarization directions.

43. The apparatus of claim 27, wherein said depth measurement means further comprises:

(i) analog to digital converting means, coupled to said sensor means, for converting sensed images into digital signals on a pixel by pixel basis; and

(ii) convolving means, coupled to said analog to digital converting means, for convolving said digital signals on a pixel by pixel basis to derive power measurement signals that correspond to the fundamental frequency of said illumination pattern at each of said pixels for each sensed scene image.

44. The apparatus of claim 43, wherein said depth measurement means further comprises:

(iii) registration correction means, coupled to said convolving means, for correcting said power measurement signals for mis-registration on a pixel by pixel basis, such that any errors introduced into said power measurement signals because of misalignment between said sensing pixels of said grid and said illumination pattern is corrected.

45. The apparatus of claim 44, wherein said registration correction means further include arithmetic means for multiplying each of said power measurement signals, on a pixel by pixel basis, by the sum of the squares of said power measurement signal'"'"'s four neighboring power measurement signals.

46. The apparatus of claim 43, wherein said depth measurement means further comprises:

(iii) normalizing means, coupled to said convolving means, for normalizing said power measurement signals on a pixel by pixel basis.

47. The apparatus of claim 43, wherein said depth measurement means further comprises:

(iii) comparator means, coupled to said convolving means, for comparing said power measurement signals for one of said sensed images, on a pixel by pixel basis, with determined power measurements for a second of said sensed images, to determine said depth information at each of said pixels.

48. The apparatus of claim 47, wherein said comparator means includes a look-up table.

49. The apparatus of claim 27, wherein said scene recovery means comprises depth map storage means, coupled to said depth measurement means, for storing derived pixel by pixel depth information for said scene as a depth map.

50. The apparatus of claim 49, further comprising:

(e) display means, coupled to said scene recovery means, for displaying said depth map as a wireframe on a bitmapped workstation.

51. The apparatus of claim 35, wherein said scene recovery means comprises three-dimensional texturemap storage means, coupled to said depth measurement means and said brightness sensor, for storing derived pixel by pixel depth information and brightness information for said scene, further comprising:

(e) display means, coupled to said scene recovery means, for displaying said three-dimensional texturemap as a wireframe on a bitmapped workstation.