Image depth perception device

• US 20140146136A1
• Filed: 09/29/2013
• Published: 05/29/2014
• Est. Priority Date: 11/27/2012
• Status: Active Grant

• Current Assignee: Chenyang Ge, Yanhui Zhou
• Original Assignee: Chenyang Ge, Yanhui Zhou
• Inventors: Ge, Chenyang, Zhou, Yanhui
• Granted Patent: US 9,995,578 B2
• Time in Patent Office: Days
• US Class Current: 348/43
• CPC Class Codes: G01C 3/32 by focusing the object, e.g. on a ground glass screen
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/579 from motion

Abstract

In view of the active vision model based on structured light, a hardware structure of a depth perception device (a chip or an IP core) for high-precision images is disclosed. Simultaneously, the module is not only capable of serving as an independent chip, but also an embedded IP core in application. Main principle of the module is as follows. Speckle image sequence (obtained from an external image sensor and unknown depth information) is processed by adaptive and uniform pre-processing sub-module, then is inputted to the module to be compared with the standard speckle image (known depth information), then motion-vector information of the inputted speckle image is obtained by pattern matching of image blocks (similarity calculation) by the block-matching motion estimation sub-module, then depth image is obtained by depth calculation, and finally high-resolution sequence of depth image is outputted by post-processing the depth image.

Claims

1. An image depth perception device, comprising:

an image adaptive pre-processing sub-module, a block-matching motion estimation sub-module, a depth calculating sub-module and a memory,

wherein an inputted speckle image sequence obtained and collected by an external image sensor is outputted to the image adaptive pre-processing sub-module in a digital video format;

the image adaptive pre-processing sub-module processes adaptive and uniformpre-processing on the inputted speckle image sequence with different characteristics of brightness, signal to noise ratio and size;

a standard speckle image with known depth and range information, which is processed by pre-processing, is permanently stored in the memory;

the block-matching motion estimation sub-module compares each inputted speckle image in the inputted speckle image sequence processed by pre-processing with the standard speckle image, and calculates displacement of image blocks in the inputted speckle image; and

according to the displacement of the image blocks in the inputted speckle image, the depth calculating sub-module calculates depth information thereof, so as to obtain a calculated depth image sequence.

2. The device, as recited in claim 1, wherein the digital video format is Bayer, or ITU656/ITU601 video format;

and

the device further comprises an image depth perception module comprising a depth image post-processing sub-module which decreases noises by a method of noise-reducing filter, so as to output a high-resolution depth image sequence.

3. The device, as recited in claim 1, wherein a process of the pre-processing of the image adaptive pre-processing sub-module comprises steps of:

converting the inputted speckle image sequence and the standard speckle image to RGB format,

then converting to YUV format via a color space conversion, and

finally adaptively de-noising and enhancing on images of the YUV format.

4. The device, as recited in claim 1, wherein working process of the block-matching motion estimation sub-module comprises steps of:

extracting an image block block<sub>m×

n</sub> with a size of m×

n from the inputted speckle image,

in the standard speckle image and centered on a corresponding position of the image block block<sub>m×

n</sub> in a search block search_block<sub>M×

N</sub> with a size of M×

N, seeking an optimum matching block of the image block block<sub>m×

n</sub> by a microprocessor according to a search strategy and a similarity measurement indicator, wherein M, N, m and n are all integers, M>

m, N>

n, in such a manner that displacement of the image block block<sub>m×

n</sub> is obtained, i.e., motion vector.

5. The device, as recited in claim 2, wherein in the noise-reducing filter method, filter thereof is a median filter method or an average filter method.

6. The device, as recited in claim 4, wherein the depth calculating sub-module compares the speckle image with the standard speckle image, calculates displacement of the image blocks in the inputted speckle image, under a circumstance that the displacement of the image blocks, a focal length of an image sensor, a dot pitch parameter of pixels of the image sensor and a reference range of the standard speckle image are known, depth value corresponding to the image block is calculated by utilizing principle of laser triangulation measurement, then corresponding depth image of the inputted speckle image is generated by calculating the depth value of the image blocks in the inputted speckle image.

7. The device, as recited in claim 4, wherein the depth calculating sub-module calculates depth of the image blocks in the inputted speckle image by a lookup table, wherein the lookup table is established according to different displacement values of the standard speckle image and corresponding depth value thereof.

8. The device, as recited in claim 4, wherein the search strategy comprises:

moving the image block in a horizontal direction, adding number of lines in a vertical direction, and searching the matching blocks one by one.

9. The device, as recited in claim 7, further comprising a microprocessor, wherein the microprocessor not only reads the standard speckle image in the memory in a way of serial port or parallel port, but also processes value estimation of initialization parameter and parameter configuration of control registers on the image sensor and each sub-module mentioned above in a way of an internal control bus (IBC) or an IIC bus, which comprises:

processing output format controlling on the image sensor, processing algorithm performance configuration, threshold setting and switch control on the image adaptive pre-processing sub-module, and processing size control of the image blocks and search window on the block-matching motion estimation sub-module, processing dynamic allocation on the lookup table of the depth calculating sub-module, and processing filter method selecting, threshold setting and switch controlling on the depth image post-processing sub-module.

10. The device, as recited in claim 8, a step length of the image blocks extracted from the inputted speckle image is less than a size of the matching block thereof.