Depth mapping using time-coded illumination

• US 9,066,087 B2
• Filed: 11/17/2011
• Issued: 06/23/2015
• Est. Priority Date: 11/19/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Shpunt, Alexander
• Primary Examiner(s): HOGUE, DENNIS A
• Application Number:
  US13/885,688
• Publication Number:
  US 20140022348A1
• Time in Patent Office: 1,314 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2527 with phase change by in-plane movement of the patern
• CPC Class Codes: G02B 7/40 using time delay of the reflected waves, e.g. of ultrasonic waves
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 25/745 Circuitry for generating timing or clock signals
• CPC Class Codes: H04N 25/771 comprising storage means other than floating diffusion

Abstract

A method for depth mapping includes illuminating an object with a time-coded pattern and capturing images of the time-coded pattern on the object using a matrix of detector elements. The time-coded pattern in the captured images is decoded using processing circuitry embedded in each of the detector elements so as to generate respective digital shift values, which are converted into depth coordinates.

Claims

1. A method for depth mapping, comprising:

illuminating an object with a time-coded pattern;

capturing images of the time-coded pattern on the object using a matrix of detector elements;

storing reference values corresponding to a reference image of the time-coded pattern at a predefined reference frame;

decoding the time-coded pattern in the captured images using processing circuitry embedded in each of the detector elements by generating respective output words from the detector elements based on the captured images, and taking differences by subtraction between the respective output words and the stored reference values so as to generate respective digital shift values; and

converting the shift values into depth coordinates.

2. The method according to claim 1, wherein illuminating the object comprises projecting a temporal sequence of spatial patterns of light onto the object.

3. The method according to claim 2, wherein the spatial patterns comprise alternating bright and dark stripes, having a different, respective spatial frequency in each of the spatial patterns.

4. The method according to claim 2, wherein the spatial patterns comprise one or more gray-scale patterns.

5. The method according to claim 2, wherein capturing the images comprises capturing multiple successive image frames, and wherein the temporal sequence extends over a given number of the successive image frames, and wherein decoding the time-coded pattern comprises combining signals from the given number of the successive image frames.

6. The method according to claim 2, wherein projecting the temporal sequence comprises:

scanning an illumination beam along a first axis, while modulating the beam; and

applying a cylindrical optical element to generate the spatial patterns by spreading the modulated beam along a second axis, perpendicular to the first axis.

7. The method according to claim 2, wherein projecting the temporal sequence comprises:

providing a patterned element comprising multiple areas, which when illuminated, created respective ones of the spatial patterns; and

illuminating the areas sequentially so as to project the temporal sequence of the spatial patterns.

8. The method according to claim 1, wherein capturing the images comprises generating electrical charge in each of the detector elements responsively to the time-coded pattern, and

wherein decoding the pattern comprises switching the charge among different charge storage components in each of the detector elements in synchronization with the timecoded pattern.

9. The method according to claim 8, wherein the temporal sequence and switching the charge are synchronized by a global clock, and wherein decoding the pattern comprises reading data out of the matrix of the detector elements under control of a local clock, which is unsynchronized with the global clock.

10. The method according to claim 9, wherein the matrix comprises multiple rows of the detector elements, and wherein reading the data out comprises outputting the data from the detector elements row by row using a rolling shutter controlled by the local clock.

11. The method according to claim 8, wherein decoding the pattern comprises:

making a comparison of the charge stored in each of the charge storage components to a threshold level;

generating bit values responsively to the comparison; and

concatenating the bit values to generate the respective digital shift values.

12. The method according to claim 8, wherein decoding the pattern comprises combining signals from the charge storage components in groups of neighboring detector elements so as to provide the depth coordinates with enhanced depth resolution.

13. The method according to claim 1, wherein illuminating the object comprises projecting a temporal sequence of spatial patterns of light onto the object, and

wherein capturing the images comprises capturing multiple successive image frames, and the temporal sequence extends over a given number of the successive image frames, and

wherein decoding the pattern comprises combining signals from the given number of the successive image frames while combining the signals from groups of neighboring detector elements.

14. The method according to claim 1, wherein converting the shift values comprises generating a depth map of the object by triangulation of the digital shift values.

15. Apparatus for depth mapping, comprising:

an illumination subassembly, which is configured to illuminate an object with a time-coded pattern;

an image capture subassembly, which comprises;

a matrix of detector elements configured to capture images of the time-coded pattern on the object;

registers configured to store reference values corresponding to a reference image of the time-coded pattern at a predefined reference frame;

processing circuitry, which is embedded in each of the detector elements and is configured to decode the time-coded pattern in the captured images so as to provide respective output words from the detector elements based on the captured images; and

a subtracter, which is coupled to take differences between the respective output words and the stored reference values so as to output respective digital shift values; and

a processor, which is configured to convert the shift values into depth coordinates.

16. The apparatus according to claim 15, wherein the illumination subassembly is configured to illuminate the object by projecting a temporal sequence of spatial patterns of light onto the object.

17. The apparatus according to claim 16, wherein the spatial patterns comprise alternating bright and dark stripes, having a different, respective spatial frequency in each of the spatial patterns.

18. The apparatus according to claim 16, wherein the spatial patterns comprise one or more grayscale patterns.

19. The apparatus according to claim 16, wherein the image capture assembly is configured to capture multiple successive image frames, and wherein the temporal sequence extends over a given number of the successive image frames, and wherein the processing circuitry is configured to combine the signals from the given number of the successive image frames to generate the shift values.

20. The apparatus according to claim 16, wherein the illumination subassembly comprises:

a radiation source, which is configured to generate an illumination beam;

scanning optics, which are configured to scan the illumination beam along a first axis while the illumination beam is modulated; and

a cylindrical optical element, which is configured to generate the spatial patterns by spreading the modulated beam along a second axis, perpendicular to the first axis.

21. The apparatus according to claim 16, wherein the illumination subassembly comprises:

a patterned element comprising multiple areas, which when illuminated, create respective ones of the spatial patterns; and

at least one radiation source, which is configured to illuminate the areas sequentially so as to project the temporal sequence of the spatial patterns.

22. The method according to claim 15, wherein the detector elements are configured to generate electrical charge responsively to the time-coded pattern, and

wherein the processing circuitry comprises, in each of the detector elements:

multiple charge storage components; and

a switch which is configured to switch the charge from the detector elements among the charge storage components in synchronization with the time-coded pattern.

23. The apparatus according to claim 22, wherein the temporal sequence and switching the charge are synchronized by a global clock, and wherein the image capture subassembly is configured to read data out of the matrix of the detector elements under control of a local clock, which is unsynchronized with the global clock.

24. The apparatus according to claim 23, wherein the matrix comprises multiple rows of the detector elements, and wherein the data from the detector elements are read out row by row using a rolling shutter controlled by the local clock.

25. The apparatus according to claim 22, wherein the processing circuitry comprises comparator logic, which is configured to make a comparison of the charge stored in each of the charge storage components to a threshold level, to generate bit values responsively to the comparison, and to concatenate the bit values to generate the respective digital shift values.

26. The apparatus according to claim 22, wherein the processing circuitry is configured to combine the signals from the charge storage components in groups of neighboring detector elements so as to provide the depth coordinates with enhanced depth resolution.

27. The method according to claim 15, wherein the illumination subassembly is configured to illuminate the object by projecting a temporal sequence of spatial patterns of light onto the object, and

wherein the image capture assembly is configured to capture multiple successive image frames, and the temporal sequence extends over a given number of the successive image frames, and

wherein the processor is configured to combine the signals from the given number of the successive image frames while combining the signals from groups of neighboring detector elements.

28. The method according to claim 15, wherein the processor is configured to convert the shift values comprises generating a depth map of the object by triangulation of the digital shift values.