Memory-efficient coded light error correction

• US 9,635,339 B2
• Filed: 08/14/2015
• Issued: 04/25/2017
• Est. Priority Date: 08/14/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Campbell, Shandon, Verrall, Stephen Michael, Atanassov, Kalin Mitkov, Miclea, Ovidiu Cristian
• Primary Examiner(s): CHAUDRY, MUJTABA M
• Application Number:
  US14/827,088
• Publication Number:
  US 20170048507A1
• Time in Patent Office: 620 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Systems and methods for correcting errors in a depth map generated by a structured light system are disclosed. In one aspect, a method includes receiving valid and invalid codewords, the valid spatial codewords included in a codebook. The method includes detecting the invalid codeword. The method includes retrieving a set of candidate valid codewords a lowest Hamming distance between the invalid codeword and the valid codewords in the codebook. The method includes estimating a median depth of neighboring locations of the invalid codeword. The method includes associating a depth with each candidate codeword and selecting the candidate with an associated depth closest to the depth estimate. The method includes assigning the depth associated with the selected candidate codeword to the location of the invalid codeword.

Claims

1. A method for correcting errors in a depth map generated by a structured light system, the method comprising:

receiving a plurality of valid codewords and at least one invalid codeword, each of the plurality of valid codewords included in a codebook of valid codewords, the codebook not including the at least one invalid codeword, each codeword representing at least one row and at least one column of pixels of a structured light image, and each codeword corresponding to a location in a depth map;

detecting the invalid codeword;

retrieving, from a memory unit, a set of candidate codewords that are included in the codebook, each candidate codeword a first Hamming distance from the invalid codeword, the first Hamming distance being a lowest Hamming distance between the at least one invalid codeword and the codewords in the codebook;

determining a depth estimate for a plurality of locations of the depth map that include at least one location adjacent to the invalid codeword location in the depth map;

associating a depth with each of the candidate codewords;

selecting the candidate codeword having an associated depth closest to the determined depth estimate; and

assigning the depth associated with the selected candidate codeword to the location of the invalid codeword.

2. The method of claim 1, wherein each pixel in the at least one column of pixels represents received light energy, and wherein each pixel n in the at least one column of pixels has a binary value.

3. The method of claim 1, wherein detecting the at least one invalid codeword comprises retrieving from the memory unit an indication that the invalid codeword is not included in the codebook.

4. The method of claim 1, further comprising populating the memory unit with an indication of whether each received plurality of codewords and the at least one invalid codeword is included in the codebook.

5. The method of claim 1, wherein the first Hamming distance between the at least one invalid codeword and each of the candidate codewords is the number of the at least one row and the at least one column of pixels with different binary values.

6. The method of claim 1, further comprising populating the memory unit with the set of candidate codewords at the first Hamming distance from the invalid codeword, the first Hamming distance being the lowest Hamming distance between the invalid codeword and codewords in the codebook.

7. The method of claim 6, further comprising populating the memory unit with a second set of candidate codewords, the second set of candidate codewords corresponding to codewords that are included in the codebook, each of the corresponding second set of candidate codewords a second Hamming distance from the invalid codeword, the second Hamming distance higher than the first Hamming distance.

8. The method of claim 1, wherein determining the depth estimate comprises computing a median depth or an average depth of the plurality of locations of the depth map.

9. The method of claim 1, wherein the plurality of locations of the depth map includes locations less than or equal to a threshold distance in locations from the location of the invalid codeword.

10. The method of claim 9, wherein the threshold distance is 1, √

{square root over (2)}, 2, √

{square root over (5)}, 2, or 2√

{square root over (2)}, corresponding to 4, 8, 12, 20, and 24 locations in the plurality of locations of the depth map, respectively.

11. An apparatus for correcting errors in a depth map generated by a structured light system, the apparatus comprising:

a memory unit configured to;

store a plurality of valid codewords and at least one invalid codeword, each of the plurality of valid codewords included in a codebook of valid codewords, the codebook not including the at least one invalid codeword, each codeword representing at least one row and at least one column of pixels of a structured light image, and each codeword corresponding to a location in a depth map,

the memory unit further configured to store a set of candidate codewords that are included in the codebook, each candidate codeword a first Hamming distance from the invalid codeword, the first Hamming distance being a lowest Hamming distance between the invalid codeword and codewords in the codebook; and

a processor in communication with the memory unit, the processor configured to;

receive the plurality of valid codewords and the at least one invalid codeword,

detect the invalid codeword,

retrieve the set of candidate codewords,

determine a depth estimate for a plurality of locations of the depth map, the plurality of locations of the depth map including at least one location adjacent to the location of the invalid codeword,

associate a depth with each of the candidate codewords,

select the candidate codeword having an associated depth closest to the depth estimate, and

assign the depth associated with the selected candidate codeword to the location of the invalid codeword.

12. The apparatus of claim 11, wherein each pixel represents received light energy, and wherein each pixel has a binary value.

13. The apparatus of claim 11, wherein the processor detects the invalid codeword by retrieving, from the memory unit, an indication that the invalid codeword is not included in the codebook.

14. The apparatus of claim 11, the processor further configured to populate the memory unit with an indication of whether each received valid or invalid codeword is included in the codebook.

15. The apparatus of claim 11, wherein the first Hamming distance between the invalid codeword and each of the candidate codewords is the number of the at least one row and the at least one column of pixels with different binary values.

16. The apparatus of claim 11, the processor further configured to populate the memory unit with the set of candidate codewords at the first Hamming distance from the invalid codeword, the first Hamming distance being the lowest Hamming distance between the invalid codeword and codewords in the codebook.

17. The apparatus of claim 16, the processor further configured to populate the memory unit with a second set of candidate codewords, the second set of candidate codewords corresponding to codewords that are included in the codebook, each of the corresponding second set of candidate codewords a second Hamming distance from the invalid codeword, the second Hamming distance higher than the first Hamming distance.

18. The apparatus of claim 11, wherein the processor is configured to estimate the depth estimate by computing a median depth or an average depth of the plurality of locations of the depth map.

19. The apparatus of claim 11, wherein the plurality of locations of the depth map includes locations less than or equal to a threshold distance in locations from the location of the invalid codeword.

20. The apparatus of claim 19, wherein the threshold distance is 1, √

{square root over (2)}, 2, √

{square root over (5)}, 2, or 2√

{square root over (2)}, corresponding to 4, 8, 12, 20, and 24 locations in the plurality of locations of the depth map, respectively.

21. An apparatus for correcting errors in a depth map generated by a structured light system, the apparatus comprising:

means for storing a plurality of valid codewords and at least one invalid codeword, each of the plurality of valid codewords included in a codebook of valid codewords, the codebook not including the at least one invalid codeword, each codeword representing at least one row and at least one column of pixels of a structured light image, and each codeword corresponding to a location in a depth map;

means for receiving a plurality of valid codewords and at least one invalid codeword,

means for detecting the invalid codeword;

means for retrieving, from the means for storing, a set of candidate codewords that are included in the codebook, each candidate codeword a first Hamming distance from the invalid codeword, the first Hamming distance being a lowest Hamming distance between the at least one invalid codeword and the codewords in the codebook;

means for determining a depth estimate for a plurality of locations of the depth map that include at least one location adjacent to the invalid codeword location in the depth map;

means for associating a depth with each of the candidate codewords;

means for selecting the candidate codeword having an associated depth closest to the determined depth estimate; and

means for assigning the depth associated with the selected candidate codeword to the location of the invalid codeword.

22. The apparatus of claim 21, wherein the means comprises a memory unit, wherein the means for receiving comprises a receiver having a processor.

23. The apparatus of claim 21, wherein each pixel represents received light energy, wherein each pixel is represented by a binary value.

24. The apparatus of claim 21, wherein the means for detecting detects the invalid codeword by retrieving, from the storing means, an indication that the invalid codeword is not included in the codebook.

25. The apparatus of claim 21, further comprising means for populating the means for storing with an indication of whether each received codeword is included in the codebook.

26. The apparatus of claim 21, further comprising means for populating the storing means with the set of candidate codewords at the first Hamming distance from the invalid codeword.

27. The apparatus of claim 26, further comprising means for populating the storing means with a second set of candidate codewords, the second set of candidate codewords corresponding to codewords that are included in the codebook, each of the corresponding second set of candidate codewords a second Hamming distance from the invalid codeword, the second Hamming distance higher than the first Hamming distance.

28. The apparatus of claim 21, wherein the means for determining is configured to estimate the depth estimate by computing a median depth or an average depth of the plurality of locations of the depth map.

29. The apparatus of claim 21, wherein the plurality of locations of the depth map includes locations less than or equal to a threshold distance in locations from the location of the invalid codewords, wherein the threshold distance is 1, √

{square root over (2)}, 2, √

{square root over (5)}, 2, or 2√

{square root over (2)}, corresponding to 4, 8, 12, 20, and 24 locations in the plurality of locations of the depth map, respectively.

30. A non-transitory computer-readable medium storing instructions for correcting errors in a depth map generated by a structured light system, the instructions when executed that, when executed, perform a method comprising:

receiving a plurality of valid codewords and at least one invalid codeword, each of the plurality of valid codewords included in a codebook of valid codewords, the codebook not including the at least one invalid codeword, and each codeword corresponding to a location in a depth map;

detecting the invalid codeword;

retrieving, from a memory unit, a set of candidate codewords that are included in the codebook, each candidate codeword a first Hamming distance from the invalid codeword, the first Hamming distance being a lowest Hamming distance between the invalid codeword and codewords in the codebook;

determining a depth estimate for a plurality of locations of the depth map, the plurality of locations of the depth map including at least one location adjacent to the location of the invalid codeword;

associating a depth with each of the candidate codewords;

selecting the candidate codeword having an associated depth closest to the depth estimate; and

assigning the depth associated with the selected candidate codeword to the location of the invalid codeword.