Method for generating a depth map, related system and computer program product

• US 9,769,454 B2
• Filed: 03/30/2015
• Issued: 09/19/2017
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Original Assignee: STMicroelectronics SRL (STMicroelectronics NV)
• Inventors: Visentini, Giovanni, Piccinelli, Emiliano Mario
• Primary Examiner(s): HESS, DANIEL A
• Application Number:
  US14/672,911
• Publication Number:
  US 20150371394A1
• Time in Patent Office: 904 Days
• Field of Search: 235454, 348 46, 348 47, 382107, 382209, 382106, 382123
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A pattern of symbols is generated and sent to a projector, wherein the pattern includes an array of symbols having a given number of symbol columns and symbol rows, and an image is obtained from a camera. Next the image is decoded in order to generate a decoded pattern of symbols and the depth map is generated as a function of the pattern and the decoded pattern. The image is decoded by placing an array of classification windows on the image and determining the displacement of each classification window in order to optimize a given cost function. Finally, the decoded pattern is generated by determining a respective symbol for each classification window.

Claims

1. A method, comprising:

generating a pattern of symbols to project as a projected image, said pattern including an array of symbols having a number of symbol columns and a number of symbol rows;

obtaining an image from a camera;

decoding said obtained image, generating a decoded pattern of symbols comprising an array having said number of symbol columns and number of symbol rows, the decoding including;

defining an array of classification windows of said obtained image, wherein said array of classification windows has said number of symbol columns and number of symbol rows;

determining a displacement of each classification window by optimizing a cost function; and

generating said decoded pattern by determining a respective symbol for each classification window; and

generating a depth map as a function of said pattern of symbols to project and said decoded pattern, wherein said determining a respective symbol for each classification window comprises classifying the symbol in each classification window using an artificial neural network.

2. The method of claim 1 wherein said array of symbols of the pattern of symbols to project is filled with at least one De Bruijn sequence of unique sub-sequences of symbols with a sub-sequence length.

3. The method of claim 2 wherein said array of symbols of the pattern of symbols to project is filled with at least one De Bruijn sequence of unique sub-sequences of symbols with a sub-sequence length by repeating the same De Bruijn sequence and adding a variable offset to said Bruijn sequence after one or more threshold numbers of symbols.

4. The method of claim 1 wherein said array of symbols of the pattern of symbols to project is filled based on an alphabet comprising only two different symbols.

5. The method of claim 1, wherein said array of symbols of the pattern of symbols to project is filled based on an alphabet comprising symbols with the same dimension, pixels of a symbol of the alphabet have a same grayscale value, and each symbol of said alphabet has a respective different pixel greyscale value.

6. The method of claim 5 wherein said alphabet comprises only a first and a second symbol, the pixels of said first symbol are all black, and the pixels of said second symbol are all white.

7. The method of claim 5 wherein said cost function is a global cost function which takes into account:

a first cost based on whether displaced classification windows contain valid symbols; and

a second cost based on whether displaced classification windows have valid distances with respect to surrounding classification windows.

8. The method of claim 1, comprising:

subjecting said image obtained from said camera to a binarization filter in order to generate a binarized version of said obtained image and performing said decoding on said binarized version of said obtained image.

9. The method of claim 1 wherein said cost function takes into account at least one of:

a first cost based on whether displaced classification windows contain valid symbols; and

a second cost based on whether displaced classification windows have valid distances with respect to surrounding classification windows.

10. The method of claim 1 wherein said generating said depth map as a function of said pattern and said decoded pattern comprises performing a matching operation, the matching operation including at least one of:

determining for each symbol position in said decoded pattern at least one matching symbol position in said pattern of symbols to project; and

determining for each symbol position in said pattern of symbols to project at least one matching symbol position in said decoded pattern.

11. The method of claim 10 wherein said generating said depth map as a function of said pattern of symbols to project and said decoded pattern comprises determining for each symbol position a distance between said symbol position and the respective at least one matching symbol position.

12. A device, comprising:

one or more memories; and

image processing circuitry configured to;

generate, based on a received image, a decoded pattern of symbols comprising an array having a number of symbol columns and number of symbol rows, the generating including;

defining an array of classification windows of said image, said array of classification windows having said number of symbol columns and number of symbol rows;

determining a displacement of each classification window by optimizing a cost function; and

determining a respective symbol for each classification window; and

generate a depth map as a function of a projected image of a pattern of symbols and said decoded pattern, the projected image of the pattern of symbols including an array of symbols having said number of symbol columns and said number of symbol rows, wherein the image processing circuitry comprises an artificial neural network configured to determine a respective symbol for each classification window.

13. The device of claim 12 wherein said array of symbols of the projected image of the pattern of symbols includes at least one De Bruijn sequence of unique sub-sequences of symbols with a sub-sequence length.

14. The device of claim 13 wherein said array of symbols of the projected image of the pattern of symbols includes a repeating of the De Bruijn sequence with a variable offset added to said Bruijn sequence after one or more threshold numbers of symbols.

15. The device of claim 12 wherein said array of symbols of the projected image of the pattern of symbols is based on an alphabet comprising only two different symbols.

16. The device of claim 12 wherein said array of symbols of the projected image of the pattern of symbols is based on an alphabet comprising symbols with a same dimension, pixels of a symbol of the alphabet have a same grayscale value, and each symbol of said alphabet has a respective different pixel greyscale value.

17. The device of claim 16 wherein said alphabet comprises only a first and a second symbol, the pixels of said first symbol are all black, and the pixels of said second symbol are all white.

18. The device of claim 12, comprising:

a binarization filter configured to filter the received image wherein the decoded pattern of symbols is generated based on the filtered image.

19. The device of claim 12 wherein said cost function takes into at least one of:

a first cost based on whether displaced classification windows contain valid symbols; and

a second cost based on whether displaced classification windows have valid distances with respect to surrounding classification windows.

20. The device of claim 12 wherein the image processing circuitry is configured to generate the depth map by,

determining for each symbol position in said decoded pattern at least one matching symbol position in said pattern of symbols of the projected image;

and

determining for each symbol position in said pattern of symbols of the projected image at least one matching symbol position in said decoded pattern.

21. The device of claim 20 wherein the image processing circuitry is configured to generate the depth map by determining for each symbol position a distance between said symbol position and the respective at least one matching symbol position.

22. The device of claim 12 wherein said cost function is a global cost function which takes into account:

a first cost based on whether displaced classification windows contain valid symbols; and

a second cost based on whether displaced classification windows have valid distances with respect to surrounding classification windows.

23. A non-transitory computer-readable medium having contents which configure a digital image processor to perform a method, the method comprising:

generating, based on a received image, a decoded pattern of symbols comprising an array having a number of symbol columns and number of symbol rows, the generating including;

defining an array of classification windows of said image, said array of classification windows having said number of symbol columns and number of symbol rows;

determining a displacement of each classification window based on a cost function; and

determining a respective symbol for each classification window, wherein said determining a respective symbol for each classification window comprises classifying the symbol in each classification window using an artificial neural network; and

generating a depth map as a function of a projected pattern of symbols and said decoded pattern, the projected pattern of symbols including an array of symbols having said number of symbol columns and said number of symbol rows.

24. The medium of claim 23 wherein the method comprises:

projecting the projected pattern of symbols.

25. The medium of claim 24 wherein said array of symbols of the projected pattern is based on an alphabet comprising symbols with a same dimension, pixels of a symbol of the alphabet have a same grayscale value, and each symbol of said alphabet has a respective different pixel greyscale value.

26. The medium of claim 25 wherein said alphabet comprises only a first and a second symbol, the pixels of said first symbol are all black, and the pixels of said second symbol are all white.

27. A system, comprising:

a projector to project an image of a pattern of symbols, said pattern including an array of symbols having a number of symbol columns and a number of symbol rows;

an image sensor;

image processing circuitry configured to;

obtain an image from the image sensor;

decode said obtained image, generating a decoded pattern of symbols comprising an array having said number of symbol columns and number of symbol rows, the decoding including;

defining an array of classification windows of said obtained image, wherein said array of classification windows has said number of symbol columns and number of symbol rows;

determining a displacement of each classification window by optimizing a function associated with the displacement; and

generating said decoded pattern by determining a respective symbol for each classification window; and

generate a depth map as a function of said pattern of symbols of the image to project and said decoded pattern, wherein the image processing circuitry comprises an artificial neural network configured to determine a respective symbol for each classification window.

28. The system of claim 27 wherein the image processing circuitry comprises a binarization filter configured to filter the obtained image and is configured to generate the decoded pattern of symbols based on the filtered image.

29. The system of claim 27 wherein said function associated with the displacement is a global cost function which takes into account:

a first cost based on whether displaced classification windows contain valid symbols; and

a second cost based on whether displaced classification windows have valid distances with respect to surrounding classification windows.

30. The system of claim 27 wherein said array of symbols of the pattern of symbols of the image to project is filled based on an alphabet comprising symbols with a same dimension, pixels of a symbol of the alphabet have a same grayscale value, and each symbol of said alphabet has a respective different pixel greyscale value.