3D OBJECT SEGMENTATION

• US 20160196659A1
• Filed: 09/25/2015
• Published: 07/07/2016
• Est. Priority Date: 01/05/2015
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Lee, Young Hoon, Vrcelj, Bojan, Reitmayr, Gerhard
• Granted Patent: US 9,866,815 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

Disclosed are a system, apparatus, and method for 3D object segmentation within an environment. Image frames are obtained from one or more depth cameras or at different times and planar segments are extracted from data obtained from the image frames. Candidate segments that comprise a non-planar object surface are identified from the extracted planar segments. In one aspect, certain extracted planar segments are identified as comprising a non-planar object surface, and are referred to as candidate segments. Confidence of preexisting candidate segments are adjusted in response to determining correspondence with a candidate segment. In one aspect, one or more preexisting candidate segments are determined to comprise a surface of a preexisting non-planar object hypothesis. Confidence in the non-planar object hypothesis is updated in response to determining correspondence with a candidate segment.

Claims

1. A method for segmenting objects from a real-world environment, the method comprising:

extracting a first plurality of planar segments from image frame depth data;

identifying, from the extracted first plurality of planar segments, one or more first candidate segments comprising a non-planar object surface;

updating a confidence of a preexisting candidate segment in response to determining correspondence with the one or more first candidate segments;

determining the preexisting candidate segment comprises a surface of a first preexisting non-planar object hypothesis; and

updating a confidence of the first preexisting non-planar object hypothesis in response to determining correspondence with the one or more first candidate segments.

2. The method of claim 1, further comprising:

identifying, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments;

determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis;

updating a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and

adding the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

3. The method of claim 2, wherein correspondence between a respective candidate segment and the preexisting non-planar object is determined according to one or more of:

a proximity between the respective candidate segment and a center of volume of the preexisting non-planar object,

a volumetric intersection between a volume of the preexisting non-planar object and a volume extended from the respective candidate segment in a direction away from a camera, either;

along a camera principal axis, or along a normal to the respective candidate segment,

a volumetric union between two volumes, or

any combination thereof.

4. The method of claim 1, wherein the confidence in the preexisting candidate segment is updated based on one or more of:

a number of depth frames where a surface of the preexisting candidate segment was expected to be visible, and has at least one first segment correspondence,

a size of an overlap between the existing candidate segment and all the corresponding first segments,

a size and shape of the one or more first corresponding segments,

a number of depth frames where a preexisting candidate segment surface was expected to be visible, and there are no corresponding first segments, or

any combination thereof.

5. The method of claim 1, wherein the confidence in the second non-planar object hypothesis is based on one or more of:

a number of candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

confidence in candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

a size of overlap of the volumes in space defined by the candidate segments exceeding a threshold of confidence,

object volume size,

proximity of the non-planar object hypothesis to other geometric primitives extracted from the environment, wherein the geometric primitives are one or more of;

other objects, or larger planar segments.

6. The method of claim 1, further comprising:

creating a new non-planar object hypothesis comprising one or more third candidate segments having no correspondence to either;

previously determined candidate segments, or

preexisting non-planar object hypotheses.

7. The method of claim 1, wherein identifying a respective candidate segment comprises determining one or more of:

whether an absolute size of the respective candidate segment is larger than a first threshold and smaller than a second threshold,

whether a ratio between a first and second principal component of a point cloud associated with the respective segment is smaller than a third threshold,

whether an angle between a camera principal axis and a segment normal is greater than fourth threshold, or

any combination thereof.

8. The method of claim 1, wherein correspondence between two or more candidate segments is according to one or more of:

a distance between planar segments or convex hulls of the two or more candidate segments to compare,

an angle between corresponding planes of the two or more candidate segments to compare,

an overlap measure between a first segment from the two or more candidate segments and a projection of the first segment onto a plane defined by a second segment from the two or more candidate segments, or

any combination thereof.

9. The method of claim 1, further comprising:

providing, in response to determining the confidence in the second non-planar object hypothesis meets a threshold object confidence, one or both of;

a representation of a 3D boundary encompassing an object volume for the second non-planar object hypothesis, or

volume occupancy information of the second non-planar object hypothesis.

10. The method of claim 1, further comprising:

removing the preexisting candidate segment from a list of preexisting segments when confidence in the preexisting candidate segment drops below a threshold; and

removing the preexisting non-planar object hypothesis when confidence in the preexisting non-planar object hypothesis drops below a threshold.

11. The method of claim 1, further comprising:

detecting a change in a number or confidence of candidate segments comprising a surface of the preexisting non-planar object hypothesis; and

updating, in response to detecting the change, volume occupancy properties associated with the preexisting non-planar object hypothesis.

12. A machine readable non-transitory storage medium having stored therein program instructions that are executable by a processor to:

extract a first plurality of planar segments from image frame depth data;

identify, from the extracted first plurality of planar segments, one or more first candidate segments comprising a non-planar object surface;

update a confidence of a preexisting candidate segment in response to determining correspondence with the one or more first candidate segments;

determine the preexisting candidate segment comprises a surface of a first preexisting non-planar object hypothesis; and

update a confidence of the first preexisting non-planar object hypothesis in response to determining correspondence with the one or more first candidate segments.

13. The medium of claim 12, further comprising instructions to:

identify, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments;

determine correspondence between the one or more second candidate segments and a second non-planar object hypothesis;

update a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and

add the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

14. The medium of claim 13, wherein correspondence between a respective candidate segment and the preexisting non-planar object is determined according to one or more of:

a proximity between the respective candidate segment and a center of volume of the preexisting non-planar object,

a volumetric intersection between a volume of the preexisting non-planar object and a volume extended from the respective candidate segment in a direction away from a camera, either;

along a camera principal axis, or along a normal to the respective candidate segment,

a volumetric union between two volumes, or

any combination thereof.

15. The medium of claim 12, wherein the confidence in the preexisting candidate segment is updated based on one or more of:

a number of depth frames where a surface of the preexisting candidate segment was expected to be visible, and has at least one first segment correspondence,

a size of an overlap between the existing candidate segment and all the corresponding first segments,

a size and shape of the one or more first corresponding segments,

a number of depth frames where a preexisting candidate segment surface was expected to be visible, and there are no corresponding first segments, or

any combination thereof.

16. The medium of claim 12, wherein the confidence in the second non-planar object hypothesis is based on one or more of:

a number of candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

confidence in candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

a size of overlap of the volumes in space defined by the candidate segments exceeding a threshold of confidence,

object volume size,

proximity of the non-planar object hypothesis to other geometric primitives extracted from the environment, wherein the geometric primitives are one or more of;

other objects, or larger planar segments.

17. The medium of claim 12, further comprising instructions to:

create a new non-planar object hypothesis comprising one or more third candidate segments having no correspondence to either;

previously determined candidate segments, or

preexisting non-planar object hypotheses.

18. The medium of claim 12, wherein identifying a respective candidate segment comprises determining one or more of:

whether an absolute size of the respective candidate segment is larger than a first threshold and smaller than a second threshold,

whether a ratio between a first and second principal component of a point cloud associated with the respective segment is smaller than a third threshold,

whether an angle between a camera principal axis and a segment normal is greater than fourth threshold, or

any combination thereof.

19. The medium of claim 12, wherein correspondence between two or more candidate segments is according to one or more of:

a distance between planar segments or convex hulls of the two or more candidate segments to compare,

an angle between corresponding planes of the two or more candidate segments to compare,

an overlap measure between a first segment from the two or more candidate segments and a projection of the first segment onto a plane defined by a second segment from the two or more candidate segments, or

any combination thereof.

20. The medium of claim 12, further comprising instructions to:

provide, in response to determining the confidence in the second non-planar object hypothesis meets a threshold object confidence, one or both of;

a representation of a 3D boundary encompassing an object volume for the second non-planar object hypothesis, or

volume occupancy information of the second non-planar object hypothesis.

21. The medium of claim 12, further comprising instructions to:

remove the preexisting candidate segment from a list of preexisting segments when confidence in the preexisting candidate segment drops below a threshold; and

remove the preexisting non-planar object hypothesis when confidence in the preexisting non-planar object hypothesis drops below a threshold.

22. The medium of claim 12, further comprising instructions to:

detect a change in a number or confidence of candidate segments comprising a surface of the preexisting non-planar object hypothesis; and

update, in response to detecting the change, volume occupancy properties associated with the preexisting non-planar object hypothesis.

23. A device to provide 3D object detection, the device comprising:

memory; and

a processor coupled to the memory and configured to;

extract a first plurality of planar segments from image frame depth data;

identify, from the extracted first plurality of planar segments, one or more first candidate segments comprising a non-planar object surface;

update a confidence of a preexisting candidate segment in response to determining correspondence with the one or more first candidate segments;

determine the preexisting candidate segment comprises a surface of a first preexisting non-planar object hypothesis; and

update a confidence of the first preexisting non-planar object hypothesis in response to determining correspondence with the one or more first candidate segments.

24. The device of claim 23, further comprising instructions to:

identify, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments;

determine correspondence between the one or more second candidate segments and a second non-planar object hypothesis;

update a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and

add the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

25. The device of claim 23, wherein the confidence in the preexisting candidate segment is updated based on one or more of:

a number of depth frames where a surface of the preexisting candidate segment was expected to be visible, and has at least one first segment correspondence,

a size of an overlap between the existing candidate segment and all the corresponding first segments,

a size and shape of the one or more first corresponding segments,

a number of depth frames where a preexisting candidate segment surface was expected to be visible, and there are no corresponding first segments, or

any combination thereof.

26. The device of claim 23, wherein the confidence in the second non-planar object hypothesis is based on one or more of:

a number of candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

confidence in candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

a size of overlap of the volumes in space defined by the candidate segments exceeding a threshold of confidence,

object volume size,

proximity of the non-planar object hypothesis to other geometric primitives extracted from the environment, wherein the geometric primitives are one or more of;

other objects, or larger planar segments.

27. The device of claim 23, wherein correspondence between two or more candidate segments is according to one or more of:

a distance between planar segments or convex hulls of the two or more candidate segments to compare,

an angle between corresponding planes of the two or more candidate segments to compare,

an overlap measure between a first segment from the two or more candidate segments and a projection of the first segment onto a plane defined by a second segment from the two or more candidate segments, or

any combination thereof.

28. An apparatus for performing 3D object detection, the apparatus comprising:

means for identifying, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments;

means for determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis;

means for updating a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and

means for adding the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

29. The apparatus of claim 28, further comprising instructions to:

means for identifying, from the first candidate segments, one or more second candidate segments without correspondence to any preexisting candidate segments;

means for determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis;

means for updating a confidence in the second non-planar object hypothesis in response to determining correspondence between the one or more second candidate segments and a second non-planar object hypothesis; and

means for adding the one or more second candidate segments to a list of segments comprising the second non-planar object hypothesis.

30. The apparatus of claim 28,

wherein the confidence in the preexisting candidate segment is updated based on one or more of:

a number of depth frames where a surface of the preexisting candidate segment was expected to be visible, and has at least one first segment correspondence,

a size of an overlap between the existing candidate segment and all the corresponding first segments,

a size and shape of the one or more first corresponding segments,

a number of depth frames where a preexisting candidate segment surface was expected to be visible, and there are no corresponding first segments, or

any combination thereof, and

wherein the confidence in the second non-planar object hypothesis is based on one or more of;

a number of candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

confidence in candidate segments currently listed as comprising a surface of the second non-planar object hypothesis,

a size of overlap of the volumes in space defined by the candidate segments exceeding a threshold of confidence,

object volume size,

proximity of the non-planar object hypothesis to other geometric primitives extracted from the environment, wherein the geometric primitives are one or more of;

other objects, or larger planar segments.