Integrated processor for 3D mapping

• US 8,456,517 B2
• Filed: 03/04/2009
• Issued: 06/04/2013
• Est. Priority Date: 07/09/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Spektor, Evgeny, Mor, Zafrir, Rais, Dmitri
• Primary Examiner(s): LIN, WEN TAI
• Application Number:
  US12/397,362
• Publication Number:
  US 20100007717A1
• Time in Patent Office: 1,553 Days
• Field of Search: None
• US Class Current: 348/43
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

A device for processing data includes a first input port for receiving color image data from a first image sensor and a second input port for receiving depth-related image data from a second image sensor. Processing circuitry generates a depth map using the depth-related image data. At least one output port conveys the depth map and the color image data to a host computer.

Claims

1. A device for processing data, comprising:

a first input port for receiving color image data, comprising a first array of color image pixels, from a first image sensor;

a second input port for receiving depth-related image data from a second image sensor;

processing circuitry, for generating a depth map using the depth-related image data, the depth map comprising a second array of depth pixels with respective depth values, and after generating the depth values, registering the depth map with the color image data by computing different, respective shifts for the depth pixels responsively to the depth values, and applying the respective shifts to the depth pixels in the second array in order to register the depth pixels with respective, associated color image pixels in the first array; and

at least one output port for conveying the registered depth map and the color image data to a host computer.

2. The device according to claim 1, wherein the first and second input ports, the processing circuitry and the at least one output port are integrated circuit components, which are fabricated on a single semiconductor substrate.

3. The device according to claim 1, wherein the at least one output port comprises a single port and is coupled to convey the depth map and the color image data to the host computer as a single data stream in a multiplexed format.

4. The device according to claim 3, wherein the processing circuitry is configured to synchronize the depth map with the color image data in the single data stream.

5. The device according to claim 3, and comprising a third input port for receiving audio information, wherein the processing circuitry is configured to incorporate the audio information together with the depth map and the color image data into the single data stream for conveyance to the host computer via the single port.

6. The device according to claim 3, wherein the single port comprises a Universal Serial Bus (USB) port.

7. The device according to claim 1, wherein the processing circuitry is configured to compute the respective shifts incrementally, such that the shifts of the pixels within a line of the depth map are computed as a function of a preceding pixel shift.

8. The device according to claim 7, wherein the processing circuitry is configured to compute the respective shifts using an incremental fitting process.

9. The device according to claim 1, wherein the processing circuitry is configured to compute the respective shifts for the pixels on the fly, while receiving the depth values, without storing the entire depth map in the device.

10. The device according to claim 1, wherein the processing circuitry is configured to compress the depth map and the color image data for transmission to the host via the at least one output port.

11. The device according to claim 10, wherein the processing circuitry is configured to apply a process of lossless compression to the depth map and the color image data.

12. The device according to claim 1, wherein the depth-related image data comprise an image of a pattern that is projected onto an object, and wherein the processing circuitry is configured to generate the depth map by measuring shifts in the pattern relative to a reference image.

13. Apparatus for imaging, comprising:

a first image sensor, which is configured to capture and output a color image of an object, comprising a first array of color image pixels;

a second image sensor, which is configured to capture and output depth-related image data with respect to the object; and

a processing device, which comprises;

a first input port for receiving the color image from the first image sensor;

a second input port for receiving the depth-related image data from the second image sensor;

processing circuitry, for generating a depth map using the depth-related image data, the depth map comprising a second array of depth pixels with respective depth values, and after generating the depth values, registering the depth map with the color image data by computing different, respective shifts for the depth pixels responsively to the depth values, and applying the respective shifts to the depth pixels in the second array in order to register the depth pixels with respective, associated color image pixels in the first array; and

at least one output port for conveying the registered depth map and the color image data to a host computer.

14. The apparatus according to claim 13, and comprising an audio sensor, wherein the processing device comprises a third input port for receiving audio information from the audio sensor, and wherein the processing circuitry is configured to convey the audio information together with the depth map and the color image data to the host computer via the at least one output port.

15. The apparatus according to claim 13, wherein the at least one output port comprises a single port and is coupled to convey the depth map and the color image data to the host computer as a single data stream in a multiplexed format.

16. The apparatus according to claim 13, and comprising an illumination subassembly, which is configured to project a pattern onto the object, wherein the depth-related image data comprise an image of the pattern on the object, and wherein the processing circuitry is configured to generate the depth map by measuring shifts in the pattern relative to a reference image.

17. A method for processing data, comprising:

receiving color image data, comprising a first array of color image pixels, from a first image sensor via a first input port;

receiving depth-related image data from a second image sensor via a second input port;

generating a depth map using the depth-related image data, the depth map comprising a second array of depth pixels with respective depth values;

after generating the depth values, registering the depth map with the color image data by computing different, respective shifts for the depth pixels responsively to the depth values, and applying the respective shifts to the depth pixels in the second array in order to register the depth pixels with respective, associated color image pixels in the first array; and

conveying the registered depth map and the color image data via an output port to a host computer.

18. The method according to claim 17, wherein the depth map is generated by processing circuitry, and wherein the first and second input ports, the processing circuitry and the at least one output port are integrated circuit components, which are fabricated on a single semiconductor substrate.

19. The method according to claim 17, wherein the at least one output port comprises a single port and is coupled to convey the depth map and the color image data to the host computer as a single data stream in a multiplexed format.

20. The method according to claim 19, wherein conveying the depth map comprises synchronizing the depth map with the color image data in the single data stream.

21. The method according to claim 19, and comprising receiving audio information, and incorporating the audio information together with the depth map and the color image data into the single data stream for conveyance to the host computer via the single port.

22. The method according to claim 19, wherein the single port comprises a Universal Serial Bus (USB) port.

23. The method according to claim 17, wherein computing the respective shifts comprises processing the respective shifts incrementally, such that the shifts of the pixels within a line of the depth map are computed as a function of a preceding pixel shift.

24. The method according to claim 23, wherein processing the respective shifts comprises computing the respective shifts using an incremental fitting process.

25. The method according to claim 17, wherein computing the respective shifts comprises calculating the respective shifts for the pixels on the fly, while receiving the depth values, without storing the entire depth map while calculating the respective shifts.

26. The method according to claim 17, wherein conveying the depth map and the color image data comprises compressing the depth map and the color image data for transmission to the host via the at least one output port.

27. The method according to claim 26, wherein compressing the depth map and the color image data comprises applying a process of lossless compression to the depth map and the color image data.

28. The method according to claim 17, wherein the depth-related image data comprise an image of a pattern that is projected onto an object, and wherein generating the depth map comprises measuring shifts in the pattern relative to a reference image.