Active sensing spatial resolution improvement through multiple receivers and code reuse

• US 10,068,338 B2
• Filed: 03/12/2015
• Issued: 09/04/2018
• Est. Priority Date: 03/12/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Atanassov, Kalin Mitkov, Goma, Sergiu Radu
• Primary Examiner(s): Beasley, Deirdre
• Application Number:
  US14/645,923
• Publication Number:
  US 20160267671A1
• Time in Patent Office: 1,272 Days
• Field of Search: 348 47
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Structured light active sensing systems transmit and receive spatial codes to generate depth maps. Spatial codes can'"'"'t be repeated within a disparity range if they are to be uniquely identified. This results in large numbers of codes for single transmitter/single receiver systems, because reflected ray traces from two object locations may be focused onto the same location of the receiver sensor, making it impossible to determine which object location reflected the code. However, the original code location may be uniquely identified because ray traces from the two object locations that focus onto the same location of the first receiver sensor may focus onto different locations on the second receiver sensor. Described herein are active sensing systems and methods that use two receivers to uniquely identify original code positions and allow for greater code reuse.

Claims

1. An imaging apparatus for generating depth information for at least one object in a scene, comprising:

a transmitter configured to project a codemask onto an object in a scene, the codemask including at least two sets of codewords that are unique within each set, and at least two of the sets of codewords each including at least some of the same codewords as another set;

a first receiver positioned at a first distance from the transmitter, the first receiver configured to capture a first image that includes codewords that are reflected from the object that is illuminated by the projected codemask;

a second receiver positioned at a second distance from the transmitter, the second receiver configured to capture a second image that includes codewords that are reflected from the object that is illuminated by the projected codemask; and

at least one processor in communication with the first receiver and the second receiver, the at least one processor configured to obtain the first image, obtain the second image, match a first codeword that exists in the first image and in the second image to codewords in the codemask, and determine depth information of the object based on the matched codewords.

2. The imaging apparatus of claim 1, wherein at least two of the sets of codewords are identical.

3. The imaging apparatus of claim 1,

wherein the at least one processor is further configured to

match the first codeword to a set of codewords in the codemask, each codeword in the set being associated with a location in the codemask and the first codeword being associated with a location in the first image and a location in the second image;

determine a first difference between the location in the codemask of the first codewords and the location of the first codewords in the first image, and determine a second difference between the location in the codemask of the first codewords and the location of the first codewords in the second image, the first difference and the second difference each corresponding to a depth of the object; and

determine depth information of the object based on the first and second differences.

4. The imaging apparatus of claim 1,

wherein the at least one processor is further configured to

identify a first set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

identify a second set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

select a codemask location based on the first and second sets of candidate codemask locations;

determine a difference between the codemask location and the location of one of the identified codewords in the first image or the second image;

determine depth information of the at least one object based on the difference in location.

5. The imaging apparatus of claim 4, wherein the selected codemask location is within a distance tolerance of a candidate codemask location in the first set and within the distance tolerance of a candidate codemask location in the second set.

6. The imaging apparatus of claim 1, wherein the at least one processor is further configured to

determine a first set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the first image

determine a second set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the second image;

and

determine depth information of the at least one object based on the determined first and second sets of differences of the plurality of codewords.

7. The imaging apparatus of claim 1, wherein the second sensor is positioned closer to the first sensor than to the transmitter.

8. The imaging apparatus of claim 1, wherein the first sensor is between the second sensor and the transmitter.

9. The imaging apparatus of claim 1, wherein the first distance and the second distance are different.

10. The imaging apparatus of claim 1, wherein the at least one processor is further configured to obtain the codemask.

11. A method for generating a depth map, the method comprising:

projecting a codemask in a projected field-of-view from a transmitter onto an object in a scene, the codemask including at least two sets of codewords that are unique within each set, and at least two of the sets of codewords each including a portion of the same codewords as the another set;

receiving at a first receiver, positioned at a first distance from the transmitter, a first image including a first reflection of the projected codemask from the surface of the one object;

receiving at a second receiver, positioned at a second distance from the transmitter, a second image including a second reflection of the projected codemask from the surface of the one object;

identifying codewords in the first and second images; and

determining depth information of the at least one object based on the identified codewords in both of the first and second images.

12. The method of claim 11, wherein at least two of the sets of codewords are identical.

13. The method of claim 11, the method further comprising:

identifying a set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

selecting a codemask location from the set of candidate codemask locations, the selected codemask location corresponding to the one of the identified codewords in the second image;

determining a difference in location between the codemask location and the location of the one of identified codewords in the first image or the second image; and

determining depth information of the at least one object based on the difference in location.

14. The method of claim 11, the method further comprising:

identifying a first set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

identifying a second set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

selecting a codemask location based on the first and second sets of candidate codemask locations;

determining a difference in location between the codemask location and the location of the one of identified codewords in the first image or the second image; and

determining depth information of the at least one object based on the difference in location.

15. The method of claim 14, wherein the selected codemask location is within a distance tolerance of a candidate codemask location in the first set and within the distance tolerance of a candidate codemask location in the second set.

16. The method of claim 11, the method further comprising:

determining a first set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the first image;

determining a second set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the second image; and

determining depth information of the at least one object based on the said determined first and second sets of differences of the plurality of codewords.

17. The method of claim 11, wherein the second sensor is positioned closer to the first sensor than to the transmitter.

18. The method of claim 11, wherein the first sensor is between the second sensor and the transmitter.

19. The method of claim 11, wherein the first distance and the second distance are different.

20. The method of claim 11, wherein the at least one processor is further configured to obtain the codemask.

21. An apparatus for generating depth map information for at least one object in a scene, comprising:

means for projecting a codemask having at least two sets of codewords that are unique within each set, and at least two sets of the sets of codewords contain at least a portion of the same codewords;

a first means for receiving a first image including a first reflection of the projected codemask from the surface of at least one object illuminated by the projected codemask, the first means for receiving positioned at a first distance from the means for projecting;

a second means for receiving a second image including a second reflection of the projected codemask from the surface of the at least one object illuminated by the projected codemask, the second means for receiving positioned at a second distance from the means for projecting;

means for identifying codewords that are in both of the first image and the second image, and

means for determining depth information of the at least one object based on the identified codewords.

22. The apparatus of claim 21, wherein the means for projecting comprises a transmitter, wherein the first means for receiving comprises a first receiver, wherein the second means for receiving comprises a second receiver, wherein the means for identifying comprises at least one processor, and wherein the means for determining comprises at least one processor.

23. The apparatus of claim 21, the apparatus further comprising:

means for identifying a set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

means for selecting a codemask location from the set of candidate codemask locations, the selected codemask location corresponding to the one of the identified codewords in the second image;

means for determining a difference in location between the codemask location and the location of the one of identified codewords in the first image or the second image; and

means for determining depth information of the at least one object based on the difference in location.

24. The apparatus of claim 21, the apparatus further comprising:

means for identifying a first set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

means for identifying a second set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

means for selecting a codemask location based on the first and second sets of candidate codemask locations;

means for determining a difference in location between the codemask location and the location of the one of identified codewords in the first image or the second image; and

means for determining depth information of the at least one object based on the difference in location.

25. The apparatus of claim 23, wherein the selected codemask location is within a distance tolerance of a candidate codemask location in the first set and within the distance tolerance of a candidate codemask location in the second set.

26. The apparatus of claim 23, the apparatus further comprising:

means for determining a first set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the first image;

means for determining a second set of differences in the location of a plurality of codewords in the projected codemask and the location of said plurality of codewords in the second image; and

means for determining depth information of the at least one object based on the said determined first and second sets of differences of the plurality of codewords.

27. A non-transitory computer-readable medium storing instructions for generating depth information, the instructions when executed perform a process comprising:

projecting a codemask in a projected field-of-view from a transmitter, the codemask having at least two sets of codewords that are unique within each set, and at least two of the sets of codewords codes contain at least a portion of the same codewords as another set;

capturing, by a first receiver, a first image of at least one object illuminated by the projected codemask, the first receiver positioned at a first distance from the transmitter;

capturing, by a second receiver, a second image of the at least one object illuminated by the projected codemask, the second receiver positioned at a second distance from the transmitter, the first and second images depicting at least a portion of the projected codemask that is reflected from a surface of the at least one object;

identifying codewords in the first image and the second image, and

determining depth information of the at least one object based on the identified codewords that are in both the first and second images.

28. The non-transitory computer-readable medium of claim 27, the process further comprising:

identifying a set of candidate codemask locations in the projected codemask corresponding to one of the identified codewords in the first image;

selecting a codemask location from the set of candidate codemask locations, the selected codemask location corresponding to the one of the identified codewords in the second image;

determining a difference in location between the selected codemask location and the location of the one of the identified codewords in the first image or the second image; and

determining depth information of the at least one object based on the difference in location.