Image depth perception device

• US 9,995,578 B2
• Filed: 09/29/2013
• Issued: 06/12/2018
• Est. Priority Date: 11/27/2012
• Status: Active Grant

• Current Assignee: Chenyang Ge, Yanhui Zhou
• Original Assignee: Chenyang Ge, Yanhui Zhou
• Inventors: Ge, Chenyang, Zhou, Yanhui
• Primary Examiner(s): Rao, Andy
• Assistant Examiner(s): Jiang, Zaihan
• Application Number:
  US14/040,709
• Publication Number:
  US 20140146136A1
• Time in Patent Office: 1,717 Days
• Field of Search: 348 43
• CPC Class Codes: G01C 3/32 by focusing the object, e.g. on a ground glass screen
• CPC Class Codes: G06T 7/223 using block-matching
• CPC Class Codes: G06T 7/579 from motion

Abstract

In view of the active vision model based on structured light, a hardware structure of a depth perception device (a chip or an IP core) for high-precision images is disclosed. Simultaneously, the module is not only capable of serving as an independent chip, but also an embedded IP core in application. Main principle of the module is as follows. Speckle image sequence (obtained from an external image sensor and unknown depth information) is processed by adaptive and uniform pre-processing sub-module, then is inputted to the module to be compared with the standard speckle image (known depth information), then motion-vector information of the inputted speckle image is obtained by pattern matching of image blocks (similarity calculation) by the block-matching motion estimation sub-module, then depth image is obtained by depth calculation, and finally high-resolution sequence of depth image is outputted by post-processing the depth image.

Claims

1. An image depth perception device, comprising:

an image adaptive pre-processing sub-module configured;

to receive a standard speckle image having known depth and range information and an inputted speckle image sequence comprising inputted speckle images captured by an image sensor in a digital video format and having different characteristics of brightness, signal to noise ratio and size, wherein the standard speckle image is an image projected onto a plane perpendicular to the center axis of a speckle projector (z-axis) and having a known position distance, and

to perform adaptive and uniform pre-processing on the standard speckle image and the inputted speckle image sequence, wherein the adaptive and uniform pre-processing comprises (a) converting format of the standard speckle image and the inputted speckle images from a Bayer format to a RGB format, (b) converting the RGB format to a YUV format by color space conversion, (c) processing adaptive de-noising on the YUV format, and (d) processing adaptive enhancing on the YUV format, so as to improve brightness uniformity of the inputted speckle images and the signal-to-noise ratio of both the standard speckle image and the inputted speckle images;

a memory for storing the pre-processed standard speckle image;

a block-matching motion estimation sub-module configured to compare each of the inputted speckle images in the pre-processed, inputted speckle image sequence with the pre-processed standard speckle image to calculate displacement of image blocks in the pre-processed inputted speckle image, wherein the block-matching motion estimation sub-module is configured to perform the comparison and the calculation by;

extracting an image block block<sub>m×

n</sub> with a size of m×

n from the pre-processed inputted speckle image, and

seeking, in the pre-processed standard speckle image, an optimum matching block of the image block block<sub>m×

n</sub> within a search block search_block<sub>M×

N</sub> having a size of M×

N and centered on a corresponding position of the image block block<sub>m×

n</sub>, according to a search strategy and a similarity measurement indicator, such that the displacement of the image block block<sub>m×

n</sub> is obtained, wherein M, N, m and n are integers, wherein M>

m and N>

n;

a depth calculating sub-module configured to calculate depth values of the image blocks in the pre-processed inputted speckle images, using a lookup table established according to different displacement values of the standard speckle image and corresponding depth values thereof, based on the displacement of the image blocks, to obtain a calculated depth image sequence, wherein one of the depth values of the image blocks is obtained from the lookup table when one of a horizontal displacement value and a vertical displacement value serves as an input value of the lookup table; and

a depth image post-processing sub-module configured to reduce noises of the calculated depth image sequence by noise-reducing filtering.

2. The device as recited in claim 1, wherein the digital video format is a Bayer or ITU656/ITU601 video format.

3. The device as recited in claim 1, wherein the noise-reducing filtering comprises median filtering or average filtering.

4. The device as recited in claim 1, wherein the search strategy comprises searching the image blocks one by one within the search block search_block_M×

N for the optimum matching block first in a horizontal direction and then in a vertical direction.

5. The device as recited in claim 1, further comprising a microprocessor configured to control operations of the image adaptive pre-processing sub-module, the memory, the block-matching motion estimation sub-module, the depth calculating sub-module and the depth image post-processing sub-module.

6. The device as recited in claim 4, wherein the block-matching motion estimation sub-module is further configured to extract the image blocks from the inputted speckle image for matching in a step less than a size of the matching block.

7. The device as recited in claim 1, wherein the adaptively de-noising comprises window filtering or Gaussian filtering, and wherein the enhancing comprises histogram enhancing, gray linear enhancing, or binarization processing.

8. The device as recited in claim 1, wherein step length of the image block block_m×

n extracted from one of the inputted speckle images is smaller than a size of the optimum matching block.

9. The device as recited in claim 1, wherein spot pattern of each of the inputted speckle images is obtained by operations of zooming or translating spot pattern of the standard speckle image.

10. The device as recited in claim 1, wherein motion-vector information of each of the inputted speckle images is obtained by block-matching motion estimation with similarity calculation.

11. The device as recited in claim 1, wherein the depth image post-processing sub-module is configured to reduce depth holes of depth images outputted by the depth image post-processing sub-module, and wherein the depth holes are formed by mismatching during processes of block-matching motion estimation.

12. The device as recited in claim 1, wherein the standard speckle image is a single speckle image.