Systems and methods for extracting information about objects from scene information

• US 9,904,867 B2
• Filed: 01/29/2017
• Issued: 02/27/2018
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: Pointivo, Inc.
• Original Assignee: Pointivo, Inc.
• Inventors: Fathi, Habib, Serrano, Miguel M., Gross, Bradden John, Ciprari, Daniel L.
• Primary Examiner(s): Potts, Ryan P
• Application Number:
  US15/418,741
• Publication Number:
  US 20170220887A1
• Time in Patent Office: 394 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

Examples of various method and systems are provided for information extraction from scene information. 2D image information can be generated from 2D images of the scene that are overlapping at least part of one or more object(s). The 2D image information can be combined with 3D information about the scene incorporating at least part of the object(s) to generate projective geometry information. Clustered 3D information associated with the object(s) can be generated by partitioning and grouping 3D data points present in the 3D information. The clustered 3D information can be used to provide, e.g., measurement information, dimensions, geometric information, and/or topological information about the object(s). Segmented 2D information can also be generated from the 2D image information. Validated 2D and 3D information can be produced by cross-referencing between the projective geometry information, clustered 3D information, and/or segmented 2D image information, and used to label the object(s) in the scene.

Claims

1. A method of generating information about one or more objects of interest in a scene, comprising:

a. providing a plurality of overlapping 2D (two-dimensional) images of a scene, wherein;

i. the scene comprises one or more objects; and

ii. the plurality of overlapping 2D images is derived from one or more single passive image capture devices;

b. selecting an object from the scene, thereby providing a selected object of interest, wherein the selection is by a computer or a user;

c. generating 2D image information from the plurality of overlapping 2D images of the scene, wherein at least some of the plurality of overlapping 2D images;

i. include all or part of the selected object, and

ii. are overlapping as to all or part of the selected object,

 

thereby providing 2D image information incorporating all or part of the selected object;

d. providing, by a computer, 3D (three-dimensional) information generated from the scene, wherein the 3D information from the scene incorporates information about all or part of the selected object, thereby providing 3D information that incorporates all or part of the selected object;

e. generating projective geometry information by combining at least some of the 2D image information incorporating all or part of the selected object and at least some of the 3D information incorporating all or part of the selected object, thereby establishing a plurality of relationships between 3D data points derived from either or both of the 2D image information incorporating all or part of the selected object and the 3D image information incorporating all or part of the selected object, wherein one or more of;

1. measurement information,

2. dimensions, and

3. geometric information,

 

is derivable for all or part of the selected object, and wherein the plurality of relationships are established by;

i. performing an image segmentation or abstraction step on the 2D image information by partitioning and grouping image elements, thereby generating a plurality of segmented 2D image information, wherein at least some of the plurality of segmented 2D image information includes information about at least part of the selected object;

ii. performing a clustering step on the 3D information incorporating all or part of the selected object to partition and group 3D data points present in the 3D information that belong together, wherein the 3D data points are associated with all or part of the selected object, thereby generating a plurality of clustered 3D information, wherein at least some of the plurality of clustered 3D information includes information about at least part of the selected object; and

iii. performing a plurality of cross-validation steps on each of the segmented 2D image information and the clustered 3D information by processing each of the plurality of segmented 2D image information and the plurality of clustered 3D information along with the combined 2D image information and the 3D information.

2. The method of claim 1, wherein the measurement information is generated, thereby providing at least one object measured value for the selected object.

3. The method of claim 2, wherein a label is generated for the selected object, thereby providing at least one labeled and measured object.

4. The method of claim 1, wherein the measurement information is generated for at least part of the selected object, wherein:

a. the selected object has one or more dimensions of interest;

b. each of the one or more dimensions of interest has an actual measurement value; and

c. the measurement information comprises one or more derived measurement values corresponding to the one or more dimensions of interest, wherein each derived measurement value is, independently, within 5% or less of the actual measurement value for the corresponding dimension on the selected object.

5. The method of claim 1, wherein at least two selected objects of interest from the scene are provided, thereby allowing topology information to be derivable for the at least two selected objects, wherein the topology information defines relationships between all or part of each of the at least two selected objects in a 3D space associated with the at least two selected objects.

6. The method of claim 5, wherein the topology information is generated for at least part of the at least two selected objects, wherein:

a. each of the at least two selected objects, independently, comprises one or more points from which a value for a distance between each of the at least two selected objects can be generated;

b. each distance between the one or more points on the at least two selected objects, independently, has an actual distance value;

c. a distance value for the distance between at least one of the points on each of the at least two selected objects is, independently, derived; and

d. each derived distance value is, independently, within 5% or less of the actual distance value between corresponding points on each of the at least two selected objects.

7. The method of claim 5, wherein a count of each of the at least two selected objects by label or by similarities in the scene is provided.

8. The method of claim 1, wherein a label is generated for the selected object in the scene, thereby providing a labeled object.

9. The method of claim 8, wherein semantic information is generated for the labeled object, thereby providing a labeled object having semantic information associated therewith.

10. The method of claim 8, further comprising:

a. comparing the labeled object with reference information, wherein the reference information includes known measurement information; and

b. in response to the comparison, rejecting the labeled object for inclusion in an object library if the reference information indicates that the label associated with the labeled object is incorrect or accepting the labeled object for inclusion into the object library if the reference object indicates that the label associated with the labeled object is correct.

11. The method of claim 10, further comprising:

a. submitting the labeled object for verification to a human reviewer when rejected; and

b. accepting the labeled object for inclusion into the object library in response to an indication by the human reviewer that the label associated with the labeled object is correct.

12. The method of claim 10, wherein semantic information is generated for the labeled object, thereby providing a labeled object having semantic information associated therewith.

13. The method of claim 1, wherein the plurality of overlapping 2D images is generated from at least one single passive image capture device that is moving in the scene.

14. The method of claim 1, wherein the 3D information comprises point clouds derived from the plurality of overlapping 2D images of the scene.

15. The method of claim 1, wherein at least one of the one or more single passive image capture devices is a video camera.

16. The method of claim 1, wherein the 3D information is derived from point clouds generated from one or more of a time of flight imaging device, a structured light imaging device, a stereoscopic camera or a depth camera.

17. The method of claim 1, wherein the 3D image information is derived from one or more wireframes.

18. A method of generating information about one or more objects of interest in a scene comprising:

a. providing 2D (two-dimensional) image information from a plurality of overlapping 2D images of a scene, wherein at least some of the plurality of overlapping 2D images;

i. include all or part of one or more objects of interest, and

ii. are overlapping as to all or part of the one or more objects of interest,

 

thereby providing 2D image information incorporating all or part of the one or more objects of interest;

b. providing 3D (three-dimensional) information generated from the scene, wherein the 3D information from the scene incorporates information about all or part of the one or more objects of interest, thereby providing 3D information that incorporates all or part the one or more objects of interest;

c. generating projective geometry information by combining at least some of the 2D image information incorporating all or part of the one or more objects of interest and at least some of the 3D information incorporating all or part of the one or more objects of interest, thereby establishing a plurality of relationships between 3D data points derived from either or both of the 2D image information incorporating all or part of the one or more objects of interest and 3D image information incorporating all or part of the one or more objects of interest;

d. generating clustered 3D information by clustering the 3D information incorporating all or part of the one or more objects of interest to partition and group the 3D data points present in the 3D information, wherein the 3D data points are associated with all or part of the one or more objects of interest, thereby providing clustered 3D information associated with all or part of the one or more objects of interest;

e. a generating segmented 2D image information by segmenting the 2D image information incorporating all or part of the one or more objects of interest to partition and group image elements, thereby providing segmented 2D image information including information about at least part of the one or more objects of interest; and

f. generating validated 2D and 3D image information by iteratively cross-referencing the projective geometry information, the clustered 3D information and the segmented 2D image information, thereby providing validated 2D and 3D image information including information about at least part of the one or more objects of interest.

19. The method of claim 18, comprising labeling at least one object of the one or more objects of interest with a single label by merging multiple labels generated from the validated 2D and 3D image information, thereby providing at least one labeled object.

20. The method of claim 19, comprising validating the at least one labeled object using library or reference information.

21. The method of claim 19, comprising generating semantic information for one or more of the at least one labeled object.

22. The method of claim 18, wherein the 3D information is generated from the 2D image information of the scene.