Depth sensing method, device and system based on symbols array plane structured light

• US 9,829,309 B2
• Filed: 07/23/2015
• Issued: 11/28/2017
• Est. Priority Date: 12/23/2014
• Status: Active Grant

• Current Assignee: RGBDsense Information Technology Ltd.
• Original Assignee: RGBDsense Information Technology Ltd.
• Inventors: Ge, Chenyang, Zhou, Yanhui
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Adrovel, William
• Application Number:
  US14/807,433
• Publication Number:
  US 20160178355A1
• Time in Patent Office: 859 Days
• Field of Search: None
• CPC Class Codes: A61B 5/0062 Arrangements for scanning
• CPC Class Codes: A61B 5/1077 Measuring of profiles
• CPC Class Codes: G01B 11/00 Measuring arrangements characterised by the use of optical techniques
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2518 Projection by scanning of the object
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré
• CPC Class Codes: G01B 11/26 for measuring angles or tapers; for testing the alignment of axes
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

The present invention discloses a depth sensing method, device and system based on symbols array plane structured light. The coded symbols array pattern is projected by the laser pattern projector to the target object or space, and the image sensor collects and obtains the successive sequence of the encoded image of the input symbols. Firstly, the input image encoded with the symbols is decoded. The decoding process includes preprocessing, symbols location, symbols recognition and symbols correction. Secondly, the disparity of the decoded symbols are calculated by the symbols match calculation between the decoded image of the input symbols with completed symbols recognition and the decoded image of the reference symbols with the known distance. Finally the depth calculation formula is combined to generate depth point cloud information of the target object or projection space that is represented in the form of grid. The present invention can quickly obtain high-resolution and high-precision depth information of the target object or projection space in dynamic scenes, which facilitates the porting or insertion as a module into the intelligent device for 3D modeling, 3D mapping, etc.

Claims

1. A depth sensing method of symbols array plane structured light, comprising:

obtaining the encoded image of the input symbols of the symbols array plane structured light;

achieving adaptive preprocessing of the encoded image of the input symbols;

performing symbols location of the preprocessed encoded image of the input symbols to identify the location of the center of all symbols in the coded image of the input symbols;

conducting symbols recognition of the preprocessed encoded image of the input symbols, mapped as the corresponding number;

correcting identified symbols in accordance with the corresponding number to complete the decoding of the encoded image of the input symbols;

making symbols match between the corrected decoded image of the input symbols and the decoded image of the reference symbols of the known distance to generate the corresponding disparity of the symbols;

determining depth of the center of the symbols in the encoded image of the input symbols in accordance with the known distance of the decoded image of the reference symbols and the disparity,

wherein making the symbols match between the corrected decoded of the input symbols and the decoded image of the reference symbols of the known distance to generate the corresponding disparity of the symbols, and determining the depth of the center of symbols in the encoded image of the input symbols in accordance with the said known distance and the disparity, comprise;

the input symbols and their surrounding symbols forming an input symbols template, searching for an optimal match template in a search window of the decoded image of the reference symbols and obtaining the match symbols of the input symbols, so as to obtain the disparity (Δ

x, Δ

y) between the input symbols and the match symbols; and

using the disparity Δ

m (Δ

x or Δ

y) in X or Y direction, in combination with the known distance parameter d of the decoded image of the reference symbols, the baseline distance S between the laser pattern projector and the image sensor, the focal length f and the dot pitch parameter μ

of the image sensor, calculating a depth information d′

of a center o of the input symbols according to the following formula;

2. The method according to claim 1, wherein performing the symbols location of the said preprocessed encoded image of the input symbols to identify the location of the center of all symbols in the said coded image of the input symbols, comprises:

computing the location of the center of symbols one by one in accordance with the design parameters or determining that in accordance with the feature information of the symbols and by means of pattern identification.

3. The method according to claim 1, wherein the symbols recognition contains a template match method that takes the code symbols as the standard match symbols template rotating at a small angle around the center of the symbols or identifying the code symbols of the encoded image of the input symbols after linear or non-linear transformation of the symbols.

4. The method according to claim 1, wherein the mapping as the corresponding number, comprises:

correct symbols identified is mapped on a specific number N, and unidentified symbols is represented with a specific number M.

5. A method described in claim 1, wherein correcting the identified symbols in accordance with the corresponding number comprises validating the identified symbols in combination with its surrounding symbols and by means of the coding rule of the said symbols.

6. The method according to claim 5, wherein the symbols conforming to the encoding rule is identified as the correct one, while the symbols inconsistent with the coding rule is represented with the specific number M as the unidentified one for further correction, which after cyclic correction and filling generates the corrected decoded image of the input symbols.

7. The method according to claim 1, wherein the symbols match between the corrected decoded image of the input symbols and the decoded image of the reference symbols of the known distance comprises the search for the closest match symbols of the said input symbols within a certain range taking the symbols of the corresponding location of the input symbols in the decoded image of the reference symbols as the center.

8. The method according to claim 2, wherein the symbols recognition contains a template match method that takes the code symbols as the standard match symbols template rotating at a small angle around the center of the symbols or identifying the code symbols of the encoded image of the input symbols after linear or non-linear transformation of the symbols.

9. The method described in claim 4, wherein correcting the identified symbols in accordance with the corresponding number comprises validating the identified symbols in combination with its surrounding symbols and by means of the coding rule of the said symbols.

10. A depth sensing device for symbols array plane structured light, comprising:

an acquisition module acquiring the encoded image of the input symbols of the symbols array plane structured light;

a preprocessing module, used for the adaptive preprocessing of the said encoded image of the input symbols;

a symbols location module used for the symbols location of the preprocessed encoded image of the input symbols to identify the location of the center of all symbols in the encoded image of the input symbols;

a symbols recognition module, used for the symbols recognition of the preprocessed encoded image of the input symbols, mapped to the corresponding number;

a symbols correction module, used for correcting the identified symbols in accordance with the corresponding number to complete the decoding of the encoded image of the input symbols;

a symbols match module, used for the symbols match between the corrected decoded image of the input symbols and the decoded image of the reference symbols a known distance to generate a corresponding disparity of the symbols;

a depth calculation module, used for determining the depth of the center of symbols in the encoded image of the input symbols in accordance with the known distance of the decoded image of the reference symbols and the disparity,

wherein the determination module uses the disparity Δ

m (Δ

x or Δ

y) at the direction X or Y and combines with the known distance parameter d of the decoded image of the reference symbols, the baseline distance S between the laser pattern projector and the image sensor, the focal length f and the dot pitch parameter μ

of the image sensor to calculate the depth information d′

of the center o of the input symbols according to the following formula;

11. The device according to claim 10, wherein said location module is used for calculating the location of the center of the said symbols one by one according to the design parameters or determining the location in accordance with the symbols feature information and by means of pattern identification.

12. The device according to claim 11, wherein said symbols recognition module by means of template match takes the code symbols as the standard match symbols template rotating at a small angle around the center of the symbols or identifying the code symbols of the encoded image of the input symbols after linear or non-linear transformation of the symbols.

13. The device according to claim 10, wherein said symbols recognition module by means of template match takes the code symbols as the standard match symbols template rotating at a small angle around the center of the symbols or identifying the code symbols of the encoded image of the input symbols after linear or non-linear transformation of the symbols.

14. The device according to claim 10, wherein said symbols recognition module is used to map the correct symbols identified with a specific number N, and the unidentified symbols is represented with a specific number M.

15. The device according to claim 10, wherein the correction module is used to validate the identified symbols combined with its surrounding symbols by means of the encoding rule of the symbols, the symbols conforming to the said encoding rule is identified as the correct one;

while the symbols inconsistent with the coding rule is represented with the specific number M as the unidentified one for further correction, which after cyclic correction and filling generates the corrected decoded image of the input symbols.

16. The device according to claim 10, wherein the match module is used to search for the closest match symbols of the input symbols within a certain range taking the symbols of the corresponding location of the input symbols in the decoded image of the reference symbols as the center.

17. A depth sensing system based on symbols array plane structured light, comprising a laser pattern projector and a depth sensing device according to claim 10, wherein after the laser pattern projector projects coded graphics to the target object or the projection space, the depth sensing device receives the encoded image of the input symbols.