Systems and methods for generating information about a building from images of the building

• US 10,592,765 B2
• Filed: 01/19/2018
• Issued: 03/17/2020
• Est. Priority Date: 01/29/2016
• Status: Active Grant

• Current Assignee: Pointivo, Inc.
• Original Assignee: Pointivo, Inc.
• Inventors: Fathi, Habib, Serrano, Miguel M., Gross, Bradden John, Ciprari, Daniel L.
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/875,149
• Publication Number:
  US 20180225539A1
• Time in Patent Office: 788 Days
• Field of Search: None
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

Examples of various method and systems are provided for information generation from images of a building. In one example, 2D building and/or building element information can be generated from 2D images of the building that are overlapping. 3D building and building element information can be generated from the 2D building and/or building element information. The 2D image information can be combined with 3D information about the building and/or building element to generate projective geometry information. Clustered 3D information can be generated by partitioning and grouping 3D data points. An information set associated with the building and/or at least one building element can then be generated.

Claims

1. A method of generating information about a building including at least one building element of interest comprising:

a) generating two-dimensional (“

2D”

) building and building element information from a plurality of overlapping 2D images of a building including at least one building element of interest;

b) generating, by a computer, three-dimensional (“

3D”

) building and building element information from the 2D building and building element information;

c) generating, by the computer, projective geometry information by combining at least some of the 2D and 3D building and building element information, thereby establishing a plurality of relationships between 3D data points derived from either or both of the 2D and 3D building and building element information, wherein the plurality of relationships are established by;

(1) performing an image segmentation or abstraction step on 2D image information by partitioning and grouping image elements, thereby generating a plurality of segmented 2D image information, wherein at least some of the plurality of segmented 2D image information includes information about at least part of a selected object;

(2) performing a clustering step on 3D information incorporating all or part of the selected object to partition and group 3D data points present in the 3D information that belong together, wherein the 3D data points are associated with all or part of the selected object, thereby generating a plurality of clustered 3D information, wherein at least some of the plurality of clustered 3D information includes information about at least part of the selected object; and

(3) performing a plurality of cross-validation steps on each of the plurality of segmented 2D image information and the plurality of clustered 3D information by processing each of the plurality of segmented 2D image information and the plurality of clustered 3D information along with combined 2D image information and the 3D information, thereby providing;

i) 3D data points associated with the building and the at least one building element of interest; and

ii) a plurality of cross-validated 3D information associated with the building and the at least one building element of interest; and

d) generating, by the computer, an information set associated with the building and the at least one building element of interest.

2. The method of claim 1 wherein the information set comprises information about at least one building element present in the building and modification of the at least one building element over time, thereby providing information usable in a building information model (BIM) application.

3. The method of claim 1, wherein the 3D building and building element information is provided as one or more of point clouds, wireframes, CAD drawings, GeoJSON data, 3D vector models, polygon meshes, and 3D models and surfaces.

4. The method of claim 1, wherein:

a) the method tracks presence, absence, or modification of previously identified building elements over a lifecycle of the building or the at least one building element of interest; and

b) the tracking is automated or semi-automated.

5. The method of claim 1, utilized in an augmented reality environment, a virtual reality environment, or a mixed reality environment.

6. The method of claim 1, wherein one or more of:

a) measurement information;

b) dimensions;

c) geometric information; and

d) optionally, labels,

are derived for either or both of the building or the at least one building element of interest.

7. The method of claim 6, wherein the measurement information is derived and comprises a value that is within about 5% of a corresponding actual measurement value for the building or the at least one building element of interest.

8. The method of claim 6, wherein the geometric information is derived.

9. The method of claim 1, wherein semantic information is derived for the at least one building element of interest.

10. The method of claim 1, wherein geolocation information is derived for the at least one building element of interest.

11. The method of claim 1, wherein the information set includes information about at least two building elements of interest, and wherein topology information is derived for the at least two building elements of interest.

12. A method of generating information about a building and building elements of interest, comprising:

a) generating two-dimensional (“

2D”

) building and building element information from a plurality of overlapping 2D images of a building including at least one building element of interest;

b) generating, by a computer, three-dimensional (“

3D”

) building and building element information wherein at least some of the 3D building and building element information is derived from a building information model (BIM);

c) generating, by the computer, projective geometry information by combining at least some of the 2D and 3D building and building element information, thereby establishing a plurality of relationships between 3D data points derived from either or both of the 2D building and building element information and the 3D building and building element information, wherein the plurality of relationships are established by;

(1) performing an image segmentation or abstraction step on 2D image information by partitioning and grouping image elements, thereby generating a plurality of segmented 2D image information, wherein at least some of the plurality of segmented 2D image information includes information about at least part of a selected object;

(2)performing a clustering step on 3D information incorporating all or part of the selected object to partition and group 3D data points present in the 3D information that belong together, wherein the 3D data points are associated with all or part of the selected object, thereby generating a plurality of clustered 3D information, wherein at least some of the plurality of clustered 3D information includes information about at least part of the selected object; and

(3) performing a plurality of cross-validation steps on each of the plurality of segmented 2D image information and the plurality of clustered 3D information by processing each of the plurality of segmented 2D image information and the plurality of clustered 3D information along with combined 2D image information and the 3D information, thereby providing;

i) 3D data points associated with the building and the at least one building element of interest; and

ii) a plurality of cross-validated 3D information associated with the building and the at least one building element of interest; and

d) generating, by the computer, an information set associated with the building and the at least one building element of interest.

13. The method of claim 12, wherein at least some of the 3D building and building element information is provided as one or more of point clouds, wireframes, CAD drawings, GeoJSON data, 3D vector models, polygon meshes, and 3D models and surfaces.

14. The method of claim 12, wherein:

a) the method tracks presence, absence, or modification of previously identified building elements over a lifecycle of the building or the at least one building element of interest; and

b) the tracking is automated or semi-automated.

15. The method of claim 12, utilized in an augmented reality environment, a virtual reality environment, or a mixed reality environment.

16. The method of claim 12, wherein one or more of:

a) measurement information;

b) dimensions;

c) geometric information; and

d) optionally, labels,

are derived for either or both of the building or the at least one building element of interest.

17. The method of claim 16, wherein the measurement information is derived and comprises a value that is within about 5% of a corresponding actual measurement value for the building or the at least one building element of interest.

18. The method of claim 16, wherein the geometric information is derived.

19. The method of claim 12, wherein geolocation information is derived for the building or the at least one building element of interest.

20. The method of claim 12, wherein the information set includes information about at least two building elements of interest, and wherein topology information is derived for the at least two building elements of interest.