Depth-Sensing Camera System

• US 20110317005A1
• Filed: 03/12/2009
• Published: 12/29/2011
• Est. Priority Date: 03/12/2009
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Atkinson, Lee Warren
• Granted Patent: US 8,681,216 B2
• Time in Patent Office: Days
• US Class Current: 348/135
• CPC Class Codes: G01S 17/50 Systems of measurement based on relative movement of target
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A depth-sensing camera system includes a common sensor configured to record color and infrared data from a target, an infrared illuminant that projects infrared light on the target, and a control logic that switches a mode of operation of the. camera between a color mode and an infrared mode. When the camera system is in the infrared mode, the infrared illuminant operates to project the infrared light on the target and when in the color mode, the infrared illuminant is disabled. The camera system also includes a color buffer for storing color chroma and luma values and a depth buffer for storing infrared luma data.

Claims

1. A depth-sensing camera system, comprising:

a common sensor configured to record color and infrared data from a target;

an infrared illuminant that projects infrared light on the target;

a control logic that switches a mode of operation of the camera between a color mode and an infrared mode, wherein when in the infrared mode, the infrared illuminant operates to project the infrared light on the target and when in the color mode, the infrared illuminant is disabled;

a color buffer for storing color chroma and luma values; and

a depth buffer for storing infrared luma data.

2. The camera system of claim 1, further comprising a processor encoded with processing logic, the processing logic, comprising:

a color module that computes the color chroma and luma values; and

an infrared module that computes the infrared luma values.

3. The camera system of claim 2, wherein the processing logic further comprises:

an exposure/gain module that records frame rate and gain settings when the camera operates in the color mode; and

a computation module that corrects the color luma values according to the frame rate and gain settings and subtracts the color luma values from the infrared luma values to produce depth data.

4. The camera system of claim 3, wherein the processing logic further comprises:

a distance module that uses the depth data to estimate distance of the target from the camera; and

a motion module that uses the depth data and estimated distance to compute motion of the target with respect to the camera.

5. The camera system of claim 4, wherein the motion of the target is based on depth data from successive frames taken of the target.

6. The camera system of claim 5, wherein the motion module further comprises a feature extraction routine that functions to identify a feature of the target, and wherein the motion is computed among successive frames for the identified feature.

7. The camera system of claim 5, wherein the motion module further comprises a luma comparison routine that compares luma data among successive frames, and wherein an increase in luma values indicates motion of the target toward the camera.

8. The camera system of claim 5, wherein the motion module comprises a distance differential routine that estimates motion of the target based on distance changes among successive frames.

9. The camera system of claim 4, wherein the processing logic is contained in a processor within the camera.

10. The camera system of claim 4, wherein the computation, distance, and velocity modules are contained in a host processor connectable to the camera.

11. The camera system of claim 1, wherein the infrared illuminant is a flat illuminant.

12. The camera system of claim 1, wherein the infrared illuminant is a structured illuminant.

13. The camera system of claim 1, further comprising a controller separate from the camera body, wherein the controller comprises the motion and distance modules, and wherein target motion and distance are used by the controller to generate control signals for operation of a proxy of the target.

14. The camera system of claim 1, further comprising a controller separate from the camera body, wherein the controller comprises the motion and distance modules, and wherein target motion and distance are used by the controller to generate control signals for operation of a host computer.

15. A method for determining motion of an object in an image scene using a camera system, comprising the steps of:

(a) capturing, in a digital camera having a common image sensor, a color image of an object in a first color frame; and

(b) switching the common image sensor to an infrared (IR) mode and capturing a color+IR image in a first IR frame.

16. The method of claim 15, further comprising the steps of:

(c) processing color pixel data from the color image to provide color chroma and luma data;

(d) storing the color chroma and luma data;

(e) processing color+IR pixel data from the color+IR image to provide color+IR luma data;

(f) storing the color+IR luma data;

(g) subtracting the color luma data from the color+IR luma data to produce depth data;

(h) repeating steps (a)-(g) for successive color and IR frames; and

(i) determining object motion based on depth data from the first and successive color and IR frames.

17. The method if claim 16, further comprising:

detecting and recording image gain and frame rate when the camera operates in the color mode; and

correcting the color luma data according to the gain and frame rate.

18. The method of claim 16, wherein the steps of adjusting the color+IR luma data and determining object motion are completed in a device separate from the digital camera.

19. The method of claim 18, wherein the device is a host computer.

20. The method of claim 16, wherein the step of determining object motion comprises comparing depth data among the successive frames, wherein an increase in the depth data indicates motion of the object toward the digital camera.

21. The method of claim 15, wherein the camera comprises a structured IR illuminant, further comprising the step of in the IR mode, projecting a grid on the object using the structured IR illuminant, wherein the grid is distorted by the object.

22. The method of claim 21, further comprising the steps of:

detecting the distortions in the grid; and

computing distance of the object from the digital camera based on the detected distortions.

23. A computer readable medium comprising programming that when executed produces the steps of:

(a) processing color pixel data captured by a camera having a common sensor to provide color chroma and luma data from an object in a first color frame;

(b) storing the color chroma and luma data;

(c) switching the camera to an infrared (IR) mode to produce color+IR data in a first IR frame;

(d) processing color+IR pixel data to provide color+IR luma data from the object in a first IR frame;

(e) storing the color+IR data;

(f) subtracting the color luma data from the color+IR luma data to produce depth data;

(g) repeating steps (a)-(f) for successive color and IR frames; and

(h) determining object motion based on depth data from the first and successive color and IR frames.

24. The computer readable medium of claim 23, wherein the programming when executed produces the further steps of:

detecting distortions in a grid produced by an IR illuminant when operating the camera in the IR mode; and

computing distance of the object from the camera based on the detected distortions.