Photographing device and method for obtaining depth information

• US 10,560,686 B2
• Filed: 06/23/2015
• Issued: 02/11/2020
• Est. Priority Date: 06/23/2015
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Xu, Yu, Fu, Mouzheng
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US15/739,600
• Publication Number:
  US 20180184071A1
• Time in Patent Office: 1,694 Days
• Field of Search: 345419
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A photographing device and a method are provided for obtaining depth information, where a binocular manner and a structured light manner are combined, and full depth information of a target object can be obtained. The method may include: obtaining, when a distance between a target object and a photographing device is greater than an effective distance of a structured light manner, depth information of the target object by using a binocular manner; and obtaining, when the distance between the target object and the photographing device is less than an effective distance of the binocular manner, the depth information of the target object by using the structured light manner.

Claims

1. A photographing device, comprising:

a first image sensor comprising M infrared light sensing pixels and N visible light sensing pixels, wherein the first image sensor is configured to acquire an infrared light image and first a visible light image, wherein a pixel value of the first visible light image is equal to X, and wherein a pixel value of the infrared light image is equal to Y;

a second image sensor configured to acquire a second visible light image;

an infrared light source configured to project a particular pattern to a target object; and

a processor coupled to the first image sensor, the second image sensor, and the infrared light source, wherein the processor is configured to;

obtain depth information of the target object using a binocular manner when a distance between the target object and the photographing device is greater than an effective distance of a structured light manner by;

obtaining the first visible light image and the infrared light image of the target object from the first image sensor;

obtaining a first reference image based on the first visible light image and the infrared light image, wherein the first reference image has a pixel value equal to X+Y;

obtaining a second reference image of the target object using the second visible light image of the second image sensor, wherein the second reference image corresponds to the second visible light image and has a pixel value equal to X+Y; and

calculating the depth information of the target object based on the first reference image and the second reference image; and

obtain the depth information of the target object using the structured light manner when the distance between the target object and the photographing device is less than an effective distance of the binocular manner.

2. The photographing device of claim 1, wherein the processor being configured to obtain the depth information of the target object using the binocular manner, and obtain the depth information of the target object using the structured light manner comprises the processor being configured to:

attempt to obtain the depth information of the target object using the binocular manner; and

obtain the depth information of the target object using the structured light manner when the depth information of the target object cannot be obtained using the binocular manner.

3. The photographing device of claim 1, wherein the processor being configured to obtain the depth information of the target object using the binocular manner, and obtain the depth information of the target object using the structured light manner comprises the processor being configured to:

attempt to obtain the depth information of the target object using the structured light manner; and

obtain the depth information of the target object using the binocular manner when the depth information of the target object cannot be obtained using the structured light manner.

4. The photographing device of claim 1, wherein the processor being configured to obtain the depth information of the target object using the binocular manner, and obtain the depth information of the target object using the structured light manner comprises the processor being configured to:

measure the distance between the target object and the photographing device;

obtain the depth information of the target object using the binocular manner when the distance between the target object and the photographing device is greater than a first preset value; and

obtain the depth information of the target object using the structured light manner when the distance between the target object and the photographing device is less than a second preset value.

5. The photographing device of claim 1, wherein the processor being configured to obtain the depth information of the target object using the structured light manner comprises the processor being configured to:

project the particular pattern to the target object using the infrared light source;

obtain, using the first image sensor, a reference pattern that is formed using depth modulation of the target object after the particular pattern is projected to the target object; and

calculate the depth information of the target object according to the reference pattern.

6. The photographing device of claim 1, wherein M is equal to ⅓

of N.

7. The photographing device of claim 1, wherein the effective distance of the binocular manner is one meter.

8. A method for obtaining depth information, the method comprising:

obtaining depth information of a target object using a binocular manner when a distance between the target object and a photographing device is greater than an effective distance of a structured light manner, depth information of the target object using a binocular manner by;

obtaining a first visible light image and an infrared light image of the target object using a first image sensor, wherein a pixel value of the first visible light image is equal to X, and wherein a pixel value of the infrared light image is equal to Y;

obtaining a first reference image based on the first visible light image and the infrared light image, wherein the first reference image has a pixel value equal to X+Y;

obtaining a second reference image of the target object using a second visible light image obtained using a second image sensor, wherein the second reference image corresponds to the second visible light image and has a pixel value equal to X+Y; and

calculating the depth information of the target object based on the first reference image and the second reference image; and

obtaining the depth information of the target object using the structured light manner when the distance between the target object and the photographing device is less than an effective distance of the binocular manner.

9. The method of claim 8, wherein obtaining the depth information of the target object using a binocular manner, and obtaining the depth information of the target object using the structured light manner comprises:

attempting to obtain the depth information of the target object using the binocular manner; and

obtaining the depth information of the target object using the structured light manner when the depth information of the target object cannot be obtained using the binocular manner.

10. The method of claim 8, wherein obtaining the depth information of the target object using a binocular manner, and obtaining the depth information of the target object using the structured light manner comprises:

attempting to obtain the depth information of the target object using the structured light manner; and

obtaining the depth information of the target object using the binocular manner when the depth information of the target object cannot be obtained using the structured light manner.

11. The method of claim 8, wherein obtaining the depth information of the target object using the binocular manner, and obtaining the depth information of the target object using the structured light manner comprises:

measuring the distance between the target object and the photographing device;

obtaining the depth information of the target object using the binocular manner when the distance between the target object and the photographing device is greater than a first preset value; and

obtaining the depth information of target object using the structured light manner when the distance between the target object and the photographing device is less than a second preset value.

12. The method of claim 8, wherein obtaining the depth information of the target object using the structured light manner comprises:

projecting a particular pattern to the target object using an infrared light source;

obtaining, using the first image sensor, a reference pattern that is formed using depth modulation of the target object after the particular pattern is projected to the target object; and

calculating the depth information of the target object according to the reference pattern.

13. The method of claim 8, wherein the first image sensor comprises M infrared light sensing pixels and N visible light sensing pixels, and wherein M is equal to ⅓

of N.

14. The method of claim 8, wherein the effective distance of the binocular manner is one meter.

15. A photographing device, comprising:

a plurality of sensors; and

a processor coupled to the plurality of sensors and configured to;

obtain depth information of a target object using binocular manner when a distance between the target object and the photographing device is greater than an effective distance of a structured light manner by;

obtaining a first visible light image and an infrared light image of the target object using a first image sensor of the plurality of sensors, wherein a pixel value of the first visible light image is equal to X, and wherein a pixel value of the infrared light image is equal to Y;

obtaining a first reference image based on the first visible light image and the infrared light image, wherein the first reference image has a pixel value equal to X+Y;

obtaining a second reference image of the target object using a second visible light image obtained using a second image sensor of the plurality of sensors, wherein the second reference image corresponds to the second visible light image acid has a pixel value equal to X+Y; and

calculating the depth information of the target object based on the first reference image and the second reference image; and

obtain the depth information of the target object using the structured light manner when the distance between the target object and the photographing device is less than an effective distance of the binocular manner.

16. The photographing device of claim 15, wherein the processor is further configured to:

attempt to obtain the depth information of the target object using the binocular manner; and

obtain the depth information of the target object using the structured light manner when the depth information of the target object cannot be obtained using the binocular manner.

17. The photographing device of claim 15, wherein the processor is further configured to:

attempt to obtain the depth information of the target object using the structured light manner; and

obtain the depth information of the target object using the binocular manner when the depth information of the target object cannot be obtained using the structured light manner.

18. The photographing device of claim 15, wherein the photographing device further comprises a measurement device configured to measure the distance between the target object and the photographing device, wherein when the measurement device determines that the distance between the target object and the photographing device is greater than a first preset value, the processor obtains the depth information of the target object using the binocular manner, and wherein when the measurement device determines that the distance between the target object and the photographing device is less than a second preset value, the processor obtains the depth information of the target object using the structured light manner.