Systems and methods for error correction in structured light

• US 9,948,920 B2
• Filed: 08/06/2015
• Issued: 04/17/2018
• Est. Priority Date: 02/27/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Nash, James Wilson, Atanassov, Kalin Mitkov, Goma, Sergiu Radu
• Primary Examiner(s): Decady, Albert
• Assistant Examiner(s): Dorman, Christian
• Application Number:
  US14/820,419
• Publication Number:
  US 20160255332A1
• Time in Patent Office: 985 Days
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G06F 11/07 Responding to the occurrence of a fault, e.g. fault tolerance
• CPC Class Codes: G06F 11/0703 Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
• CPC Class Codes: G06F 11/0751 Error or fault detection not based on redundancy power supply failures G06F1/30; network fault management H04L41/06
• CPC Class Codes: G06F 11/0793 Remedial or corrective actions recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40
• CPC Class Codes: G06F 11/10 Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
• CPC Class Codes: H03M 13/39 Sequence estimation, i.e. using statistical methods for the reconstruction of the original codes
• CPC Class Codes: H03M 13/47 Error detection, forward error correction or error protection, not provided for in groups H03M13/01 - H03M13/37
• CPC Class Codes: H03M 13/61 Aspects and characteristics of methods and arrangements for error correction or error detection, not provided for otherwise
• CPC Class Codes: H03M 13/615 Use of computational or mathematical techniques
• CPC Class Codes: H04L 1/0045 Arrangements at the receiver end
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

Systems and methods for error correction in structured light are disclosed. In one aspect, a method includes receiving, via a receiver sensor, a structured light image of at least a portion of a composite code mask encoding a plurality of codewords, the image including an invalid codeword. The method further includes detecting the invalid codeword. The method further includes generating a plurality of candidate codewords based on the invalid codeword. The method further includes selecting one of the plurality of candidate codewords to replace the invalid codeword. The method further includes generating a depth map for an image of the scene based on the selected candidate codeword. The method further includes generating a digital representation of a scene based on the depth map. The method further includes outputting the digital representation of the scene to an output device.

Claims

1. A method of generating a digital representation of a scene, the method comprising:

receiving at a sensor a structured light image of a plurality of encoded codewords, the plurality of encoded codewords including an invalid codeword;

detecting, using a processor, the invalid codeword by determining whether each encoded codeword in the structured light image corresponds to a code in a codebook stored in memory, the codebook comprising a list of valid codes;

generating, using the processor, candidate codewords based on the invalid codeword;

selecting, using the processor, one of the candidate codewords to replace the invalid codeword, wherein said selecting one of the candidate codewords comprises

selecting from the generated candidate codewords a subset of candidate codewords, each candidate codeword in the subset being the same as one of the plurality of encoded codewords,

determining a similarity between each candidate codeword in the subset and codewords within a local neighborhood around the invalid codeword; and

selecting the candidate codeword in the subset most similar to the local neighborhood to replace the invalid codeword;

generating, using the processor, a depth map for an image of the scene based on the selected candidate codeword;

generating, using the processor, a digital representation of a scene based on the depth map; and

outputting the digital representation of the scene to an output device.

2. The method of claim 1, wherein the encoded codewords in the structured light image are formed of combinations of a plurality of basis functions and rotated versions of the plurality of basis functions.

3. The method of claim 2, wherein detecting the invalid codeword comprises comparing each encoded codeword in the structured light image to each of the plurality of basis functions and rotated versions of the plurality of basis functions.

4. The method of claim 3, wherein the comparing comprises match filtering each encoded codeword in the structured light image to each of the plurality of basis functions and rotated versions of the plurality of basis functions.

5. The method of claim 1, wherein generating the plurality of candidate codewords comprises individually perturbing each element of the invalid codeword to generate a candidate codeword for each individual perturbation.

6. The method of claim 1, wherein generating the plurality of candidate codewords comprises iteratively perturbing at least two elements of the invalid codeword to generate a candidate codeword for each iteration until all combinations of at least two elements have been perturbed once.

7. The method of claim 1, wherein the similarity corresponds to a difference in depth associated with each of the selected subset of candidate codewords and a depth associated with the local neighborhood of codewords around the invalid codeword, and wherein the local neighborhood of codewords around the invalid codeword consists of codewords within 1, 2, 3, 4, 5, or 6 codeword locations of the location of the invalid codeword.

8. The method of claim 7, wherein the depth associated with the local neighborhood is a median depth.

9. An apparatus for generating a digital representation of a scene, comprising:

a receiver sensor configured to receive a structured light image of a plurality of encoded codewords, the plurality of encoded codewords including an invalid codeword; and

a processing circuit in communication with the receiver sensor, the processing circuit configured to

detect the invalid codeword by determining whether each encoded codeword in the structured light image corresponds to a code in a codebook stored in memory, the codebook comprising a list of valid codes,

generate candidate codewords based on the invalid codeword,

select one of the candidate codewords to replace the invalid codeword, wherein said select one of the candidate codewords comprises;

selecting a subset of candidate codewords, each candidate codeword in the subset being the same as one of the plurality of encoded codewords,

determining a similarity between each candidate codeword in the subset and codewords within a local neighborhood around the invalid codeword, and

selecting the candidate codeword in the subset most similar to the local neighborhood to replace the invalid codeword,

generate a depth map for an image of the scene based on the selected candidate codeword,

generate a digital representation of a scene based on the depth map, and

output the digital representation of the scene to an output device.

10. The apparatus of claim 9, wherein the encoded codewords in the structured light image are formed of combinations of a plurality of basis functions and rotated versions of the plurality of basis functions.

11. The apparatus of claim 10, wherein the processing circuit is further configured to compare each encoded codeword in the structured light image to each of the plurality of basis functions and rotated versions of the plurality of basis functions.

12. The apparatus of claim 10, wherein the processing circuit is further configured to match filter each encoded codeword in the structured light image to each of the plurality of basis functions and rotated versions of the plurality of basis functions.

13. The apparatus of claim 9, wherein the processing circuit is further configured to perturb each element of the invalid codeword and generate a candidate codeword for each individual perturbation.

14. The apparatus of claim 13, wherein the processing circuit is further configured to iteratively perturb at least two elements of the invalid codeword to generate a candidate codeword for each iteration until all combinations of at least two elements have been perturbed once.

15. The apparatus of claim 9, wherein the similarity corresponds to a difference in depth associated with each of the selected subset of candidate codewords and a depth associated with the local neighborhood of codewords around the invalid codeword, and wherein the local neighborhood of codewords around the invalid codeword consists of codewords within 1, 2, 3, 4, 5, or 6 codeword locations around the codeword location of the invalid codeword.

16. The apparatus of claim 15, wherein the depth associated with the local neighborhood is a median depth.

17. An apparatus for generating a digital representation of a scene, comprising:

means for receiving a structured light image of a plurality of encoded codewords, the plurality of encoded codewords including an invalid codeword;

means for detecting the invalid codeword comprising means for determining whether each encoded codeword in the structured light image corresponds to a code in a codebook, the codebook comprising a list of valid codes;

means for generating candidate codewords based on the invalid codeword;

means for selecting one of the candidate codewords to replace the invalid codeword, wherein said means for selecting one of the candidate codewords to replace the invalid codeword comprises

means for selecting from the generated candidate codewords a subset of candidate codewords, each candidate codeword in the subset being the same as one of the plurality of encoded codewords,

means for determining a similarity between each candidate codeword in the subset and codewords within a local neighborhood around the invalid codeword, and

means for selecting the candidate codeword in the subset most similar to the local neighborhood to replace the invalid codeword;

means for generating a depth map for an image of the scene based on the selected candidate codeword;

means for generating a digital representation of a scene based on the depth map; and

means for outputting the digital representation of the scene to an output device.

18. The apparatus of claim 17, wherein the receiving means comprises a receiver sensor, wherein the detecting invalid codeword means comprises a processing circuit, wherein the generating candidate codeword means comprises the processing circuit, wherein the selecting codeword means comprises the processing circuit, wherein the generating depth map means comprises the processing circuit, wherein the generating digital representation means comprises the processing circuit, and wherein the outputting means comprises the processing circuit.

19. The apparatus of claim 17, wherein the encoded codewords in the structured light image are formed of combinations of a plurality of basis functions and rotated versions of the plurality of basis functions, wherein the detecting invalid codeword means comprises means for comparing each encoded codeword to each of the plurality of basis functions and rotated versions of the plurality of basis functions, and wherein the comparing means comprises means for match filtering each encoded codeword to each of the plurality of basis functions and rotated versions of the plurality of basis functions.

20. The apparatus of claim 17, wherein the generating means comprises individually perturbing each element of the invalid codeword to generate a candidate codeword for each individual perturbation, and wherein the generating means comprises iteratively perturbing at least two elements of the invalid codeword to generate a candidate codeword for each iteration until all combinations of at least two elements have been perturbed once.

21. The apparatus of claim 17, wherein the similarity corresponds to a difference in depth associated with each of the selected subset of candidate codewords and a depth associated with the local neighborhood of codewords around the invalid codeword, wherein the depth associated with the local neighborhood is a median depth and wherein the local neighborhood of codewords around the invalid codeword consists of codewords within 1, 2, 3, 4, 5, or 6 codeword locations around the location of the invalid codeword.

22. A non-transitory computer-readable medium storing instructions for generating a digital representation of a scene, the instructions that, when executed, perform a method comprising:

receiving, via a receiver sensor, a structured light image of a plurality of encoded codewords, the plurality of encoded codewords including an invalid codeword;

detecting the invalid codeword, wherein detecting the invalid codeword comprises determining whether each encoded codeword in the structured light image corresponds to a code in a codebook, the codebook comprising a list of valid codes;

generating candidate codewords based on the invalid codeword;

selecting one of the candidate codewords to replace the invalid codeword, wherein said selecting one of the candidate codewords to replace the invalid codeword comprises

selecting a subset of candidate codewords, each candidate codeword in the subset being the same as one of the encoded codewords,

determining a similarity between each candidate codeword in the subset and codewords within a local neighborhood around the invalid codeword, and

selecting the candidate codeword most similar to the local neighborhood to replace the invalid codeword;

generating a depth map for an image of the scene based on the selected candidate codeword;

generating a digital representation of a scene based on the depth map; and

outputting the digital representation of the scene to an output device.

23. The method of claim 22, wherein generating the plurality of candidate codewords comprises changing, one at a time, each bit of the invalid codeword and generating a candidate codeword for each change of a bit.