Integrated processor for 3D mapping

• US 8,848,039 B2
• Filed: 05/01/2013
• Issued: 09/30/2014
• Est. Priority Date: 07/09/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: PrimeSense Ltd (Apple Inc.)
• Inventors: Spektor, Evgeny, Mor, Zafrir, Rais, Dmitri
• Primary Examiner(s): LIN, WEN TAI
• Application Number:
  US13/874,562
• Publication Number:
  US 20130241824A1
• Time in Patent Office: 517 Days
• Field of Search: None
• US Class Current: 348/43
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects

Abstract

A device for processing data includes a first input port for receiving color image data from a first image sensor and a second input port for receiving depth-related image data from a second image sensor. Processing circuitry generates a depth map using the depth-related image data. At least one output port conveys the depth map and the color image data to a host computer.

Claims

1. A device for processing data, comprising:

a single semiconductor substrate; and

integrated circuit components, fabricated on the single semiconductor substrate, comprising;

a first input port for receiving color image data from a first sensor;

a second input port for receiving depth-related data from a second sensor, wherein the depth-related data comprise an image of a pattern that is projected onto an object;

processing circuitry, configured to generate a depth map using at least the depth-related data by measuring shifts in the pattern relative to a reference image, and to synchronize and multiplex the depth map with the color image data in a single multiplexed data stream; and

a single output port for conveying the multiplexed data stream to a host computer.

2. The device according to claim 1, and comprising a third input port for receiving audio information, wherein the processing circuitry is configured to incorporate the audio information together with the depth map and the color image data into the multiplexed data stream for conveyance to the host computer via the single output port.

3. The device according to claim 1, wherein the single output port comprises a Universal Serial Bus (USB) port.

4. The device according to claim 1, wherein the processing circuitry is configured to register the depth map with the color image data.

5. The device according to claim 4, wherein the processing circuitry is configured to take into account a distance between the first and second sensors in registering the depth map with the color image data.

6. The device according to claim 1, wherein the processing circuitry is configured to compress the depth map and the color image data for transmission to the host via the single output port.

7. The device according to claim 6, wherein the processing circuitry is configured to apply a process of lossless compression to the depth map and the color image data.

8. Apparatus for imaging, comprising:

an illumination subassembly, which is configured to project a pattern onto an object;

a first sensor, which is configured to capture and output color image data with respect to the object;

a second sensor, which is configured to capture and output depth-related data with respect to the object, wherein the depth-related data comprise an image of the pattern on the object; and

a processing device, which comprises;

a single semiconductor substrate; and

integrated circuit components, fabricated on the single semiconductor substrate, comprising;

a first input port for receiving the color image data from the first sensor;

a second input port for receiving the depth-related data from the second sensor;

processing circuitry, configured to generate a depth map using at least the depth-related data by measuring shifts in the pattern relative to a reference image, and to synchronize and multiplex the depth map with the color image data in a single multiplexed data stream; and

a single output port for conveying the depth map and the color image data to a host computer.

9. The apparatus according to claim 8, and comprising an audio sensor, wherein the processing device comprises a third input port for receiving audio information from the audio sensor, and wherein the processing circuitry is configured to convey the audio information together with the depth map and the color image data to the host computer via the single output port.

10. A method for processing data, comprising:

receiving color image data from a first sensor via a first input port of an integrated circuit device formed on a single semiconductor substrate;

receiving depth-related data from a second sensor via a second input port of the integrated circuit device, wherein the depth-related data comprise an image of a pattern that is projected onto an object;

generating, using processing circuitry in the integrated circuit device, a depth map using at least the depth-related data by measuring shifts in the pattern relative to a reference image, and synchronizing and multiplexing the depth map with the color image data in a single multiplexed data stream; and

conveying the multiplexed data stream via a single output port of the integrated circuit device to a host computer.

11. The method according to claim 10, and comprising receiving audio information, and incorporating the audio information together with the depth map and the color image data into the single multiplexed data stream for conveyance to the host computer via the single output port.

12. The method according to claim 10, wherein the single output port comprises a Universal Serial Bus (USB) port.

13. The method according to claim 10, wherein generating the depth map comprises registering the depth map with the color image data.

14. The method according to claim 13, wherein registering the depth map with the color image data takes into account a distance between the first and second sensors.

15. The method according to claim 10, wherein conveying the multiplexed data stream comprises compressing the depth map and the color image data for transmission to the host via the single output port.

16. The method according to claim 15, wherein compressing the depth map and the color image data comprises applying a process of lossless compression to the depth map and the color image data.