REFERENCE IMAGE TECHNIQUES FOR THREE-DIMENSIONAL SENSING

• US 20100225746A1
• Filed: 02/18/2010
• Published: 09/09/2010
• Est. Priority Date: 03/05/2009
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Shpunt, Alexander, Galezer, Niv, Rais, Dmitri
• Granted Patent: US 8,786,682 B2
• Time in Patent Office: Days
• US Class Current: 348/50
• CPC Class Codes: G01B 11/2518 Projection by scanning of the object
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: H04N 13/218 using spatial multiplexing
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

A method including providing a device that projects a pattern of coherent radiation. The method further includes capturing a reference image of the pattern using an image sensor by projecting the pattern of the coherent radiation onto a reference surface and engendering a relative motion between the reference surface and the image sensor while capturing the reference image. The method also includes storing the reference image in a memory associated with the device.

Claims

1. A method, comprising:

providing a device that projects a pattern of coherent radiation;

capturing a reference image of the pattern using an image sensor by projecting the pattern of the coherent radiation onto a reference surface and engendering a relative motion between the reference surface and the image sensor while capturing the reference image; and

storing the reference image in a memory associated with the device.

2. The method according to claim 1, wherein engendering the relative motion comprises connecting a motion actuator to the device and activating the motion actuator while capturing the reference image.

3. The method according to claim 1, wherein engendering the relative motion comprises connecting a motion actuator to a reference object comprising the reference surface, and activating the motion actuator while capturing the reference image.

4. The method according to claim 1, wherein engendering the relative motion comprises moving the device parallel to the reference surface.

5. The method according to claim 1, wherein engendering the relative motion comprises moving the reference surface parallel to itself.

6. The method according to claim 1, wherein engendering the relative motion comprises selecting an amplitude of the relative motion so as to average out effects of secondary speckles generated at the reference surface.

7. The method according to claim 1, wherein engendering the relative motion comprises selecting an amplitude of the relative motion in response to a resolution of the image sensor.

8. The method according to claim 1, wherein the device comprises the image sensor, the method further comprising:

capturing an alternative image of the pattern projected onto the reference surface using another image sensor external to the device;

registering the reference image and the alternative image; and

producing an enhanced reference image from the registered reference image and alternative image,

wherein storing the reference image in the memory comprises storing the enhanced reference image in the memory.

9. The method according to claim 1, wherein capturing the reference image comprises:

capturing a first image of the pattern while the reference surface is at a first distance from the device, and a second image of the pattern while the reference surface is at a second distance, different from the first distance, from the device; and

registering the first and second images to produce an alternative reference image,

wherein storing the reference image in the memory comprises storing the alternative reference image in the memory.

10. The method according to claim 1, wherein capturing the reference image comprises:

capturing a first image of the pattern while the reference surface is at a first distance from the device, and a second image of the pattern while the reference surface is at a second distance, different from the first distance, from the device, the first and the second images being configured to be registered to produce an alternative reference image; and

storing the first and second images in the memory.

11. The method according to claim 1, wherein the device is configured to generate a three-dimensional (3D) map of an object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

12. The method according to claim 11, wherein the device comprises the image sensor, and wherein capturing the test image comprises capturing the test image using the image sensor.

13. The method according to claim 1, wherein the image sensor has an integration time, and wherein the reference surface is stationary as measured with respect to the integration time.

14. A method, comprising:

providing a device that projects a pattern onto an object and captures an image of the pattern on the object using a first image sensor having a first optical characteristic;

capturing a reference image of the pattern using a second image sensor having a second optical characteristic enhanced with respect to the first optical characteristic, by projecting the pattern onto a reference surface;

registering the reference image in a frame of reference of the device; and

storing the reference image in a memory associated with the device.

15. The method according to claim 14, wherein capturing the reference image comprises capturing a first image of the pattern projected onto the reference surface using the first image sensor, and capturing a second image of the pattern projected onto the reference surface using the second image sensor, and wherein registering the reference image comprises registering the first and second images to produce a registered reference image, and wherein storing the reference image comprises storing the registered reference image in the memory.

16. The method according to claim 14, wherein the device comprises the first image sensor, and wherein the second image sensor is external to and separate from the device.

17. The method according to claim 14, wherein the pattern is projected using incoherent radiation.

18. The method according to claim 14, wherein the first and second optical characteristics comprise at least one of respective fields of view, respective resolutions, respective signal to noise ratios, and respective dynamic ranges.

19. The method according to claim 14, wherein the device is configured to generate a three-dimensional (3D) map of the object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

20. The method according to claim 14, and comprising generating a map of the object using the reference image and the image of the pattern on the object.

21. A method, comprising:

providing a device that projects a pattern onto an object;

capturing a reference image of the pattern using an image sensor by projecting the pattern onto a reference surface located at a first distance from the image sensor and at a second distance, different from the first distance, from the image sensor;

registering the reference image in a frame of reference of the device; and

storing the reference image in a memory associated with the device.

22. The method according to claim 21, wherein capturing the reference image comprises capturing a first image of the pattern projected onto the reference surface at the first distance using the image sensor, and capturing a second image of the pattern projected onto the reference surface at the second surface using the image sensor, and wherein registering the reference image comprises registering the first and second images to produce a registered reference image, and wherein storing the reference image comprises storing the registered reference image in the memory.

23. The method according to claim 21, wherein projecting the pattern comprises projecting the pattern into a first field of view, and wherein the image sensor has a second field of view different from the first field of view.

24. The method according to claim 21, wherein the device is configured to generate a three-dimensional (3D) map of the object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

25. The method according to claim 21, and comprising generating a map of the object using the reference image and the image of the pattern on the object.

26. Apparatus, comprising:

a projection and imaging device comprising;

a projector that projects a pattern of coherent radiation; and

an image sensor that captures a reference image of the pattern of coherent radiation projected onto a reference surface; and

a processor which is configured to;

implement a relative motion between the device and the reference surface while the image sensor captures the reference image, and

store the reference image in a memory associated with the device.

27. The apparatus according to claim 26, and comprising a motion actuator connected to the device, and wherein the processor is configured to activate the motion actuator while the image sensor captures the reference image.

28. The apparatus according to claim 26, and comprising a motion actuator connected to a reference object comprising the reference surface, and wherein the processor is configured to activate the motion actuator while the image sensor captures the reference image.

29. The apparatus according to claim 26, wherein implementing the relative motion comprises moving the device parallel to the reference surface.

30. The apparatus according to claim 26, wherein implementing the relative motion comprises moving the reference surface parallel to itself.

31. The apparatus according to claim 26, wherein implementing the relative motion comprises selecting an amplitude of the relative motion so as to average out effects of secondary speckles generated at the reference surface.

32. The apparatus according to claim 26, wherein implementing the relative motion comprises selecting an amplitude of the relative motion in response to a resolution of the image sensor.

33. The apparatus according to claim 26, wherein the device is configured to generate a three-dimensional (3D) map of an object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

34. The apparatus according to claim 26, wherein the image sensor has an integration time, and wherein the reference surface is stationary as measured with respect to the integration time.

35. Apparatus, comprising:

a projection and imaging device comprising;

a projector that projects a pattern of radiation onto an object; and

a first image sensor having a first optical characteristic, that captures an image of the pattern on the object;

a second image sensor having a second optical characteristic enhanced with respect to the first optical characteristic and which is configured to capture a reference image of the pattern projected onto a reference surface; and

a processor that is configured to;

register the reference image in a frame of reference of the device; and

store the registered reference image in a memory associated with the device.

36. The apparatus according to claim 35, and comprising capturing a first image of the pattern projected onto the reference surface using the first image sensor, and capturing a second image of the pattern projected onto the reference surface using the second image sensor, and wherein registering the reference image comprises registering the first and second images to produce a registered reference image, and wherein storing the reference image comprises storing the registered reference image in the memory.

37. The apparatus according to claim 35, wherein the second image sensor is external to and separate from the device.

38. The apparatus according to claim 35, wherein the projector projects the pattern using incoherent radiation.

39. The apparatus according to claim 35, wherein the second image sensor has a field of view which is larger than a field of view of the first image sensor.

40. The apparatus according to claim 35, wherein the first and second optical characteristics comprise at least one of respective fields of view, respective resolutions, respective signal to noise ratios, and respective dynamic ranges.

41. The apparatus according to claim 35, wherein the device is configured to generate a three-dimensional (3D) map of the object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

42. Apparatus, comprising:

a projection and imaging device comprising;

a projector that projects a pattern of radiation; and

an image sensor that captures a reference image of the pattern projected on a reference surface; and

a processor which is configured to;

generate the reference image by projecting the pattern onto the reference surface located at a first distance from the image sensor and at a second distance, different from the first distance, from the image sensor,

register the reference image in a frame of reference of the device, and

store the registered reference image in a memory associated with the device.

43. The apparatus according to claim 42, wherein projecting the pattern comprises projecting the pattern into a first field of view, and wherein the image sensor has a second field of view different from the first field of view.

44. The apparatus according to claim 42, wherein the device is configured to generate a three-dimensional (3D) map of an object by capturing a test image of the pattern on the object and measuring local transverse shifts of the pattern in the test image relative to the reference image.

45. A method, comprising:

providing a device that projects a pattern onto an object;

capturing a reference image of the pattern using an image sensor by projecting the pattern onto a reference surface which is oriented with respect to the image sensor to have a first region of the reference surface a first distance from the image sensor and to have a second region of the reference surface a second distance, different from the first distance, from the image sensor;

registering the reference image in a frame of reference of the device; and

storing the reference image in a memory associated with the device.

46. The method according to claim 45, wherein the reference surface is planar and is oriented non-orthogonally with respect to an optical axis of the image sensor.

47. The method according to claim 45, wherein the reference surface is curved between the first distance and the second distance.

48. The method according to claim 47, wherein a curvature of the reference surface is preconfigured to match a geometrical disparity between the image sensor and a projector configured to project the pattern, so as to introduce a constant rate of shrinkage of the reference image.

49. Apparatus, comprising:

a projection and imaging device comprising;

a projector that projects a pattern of radiation onto an object; and

an image sensor that captures a reference image of the pattern projected onto a reference surface which is oriented with respect to the image sensor to have a first region of the reference surface a first distance from the image sensor and to have a second region of the reference surface a second distance, different from the first distance, from the image sensor; and

a processor which is configured to;

use the image sensor to capture the reference image by projecting the pattern onto the reference surface,

register the reference image in a frame of reference of the device, and

store the registered reference image in a memory associated with the device.

50. The apparatus according to claim 49, wherein the reference surface is planar and is oriented non-orthogonally with respect to an optical axis of the image sensor.

51. The apparatus according to claim 49, wherein the reference surface is curved between the first distance and the second distance.

52. The apparatus according to claim 51, wherein a curvature of the reference surface is preconfigured to match a geometrical disparity between the image sensor and the projector, so as to introduce a constant rate of shrinkage of the reference image.