Context aware surface scanning and reconstruction

• US 8,965,741 B2
• Filed: 04/24/2012
• Issued: 02/24/2015
• Est. Priority Date: 04/24/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: McCulloch, Daniel J., Hastings, Ryan L., Scott, Jason, Hirzel, Holly A., Mount, Brian J.
• Primary Examiner(s): Thangavelu, Kandasamy
• Application Number:
  US13/454,258
• Publication Number:
  US 20130282345A1
• Time in Patent Office: 1,036 Days
• Field of Search: 703/6, 703/7, 345/419, 324/326, 705/7, 705/50, 280/735, 52/799, 522/22, 702/150
• US Class Current: 703/6
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06G 7/48 Analogue computers for specific processes, systems or devices, e.g. simulators
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system for generating and updating a 3D model of a structure as the structure is being constructed or modified is described. The structure may comprise a building or non-building structure such as a bridge, parking garage, or roller coaster. The 3D model may include virtual objects depicting physical components or other construction elements of the structure. Each construction element may be associated with physical location information that may be analyzed over time in order to detect movement of the construction element and to predict when movement of the construction element may cause a code or regulation to be violated. In some cases, a see-through HMD may be utilized by a construction worker while constructing or modifying a structure in order to verify that the placement of a construction element complies with various building codes or regulations in real-time.

Claims

1. A method for recording actual physical locations associated with one or more construction elements of a structure using a mobile device, comprising:

acquiring a 3D map of the structure, the 3D map includes one or more virtual objects corresponding with the one or more construction elements, the one or more construction elements include a first construction element of the structure;

identifying at the mobile device the first construction element within a field of view of the mobile device;

determining a current set of point locations associated with the first construction element in response to the identifying at the mobile device the first construction element;

updating a first virtual object of the one or more virtual objects with the current set of point locations, the first virtual object corresponds with the first construction element;

determining a confidence value associated with the current set of point locations;

displaying on the mobile device one or more images associated with the first construction element and the confidence value;

identifying a plurality of anchor points associated with the structure;

determining a plurality of locations associated with the plurality of anchor points; and

verifying a coordinate system associated with the 3D map based on the plurality of locations.

2. The method of claim 1, further comprising:

acquiring a first set of building codes associated with the structure;

verifying the first set of building codes based on the 3D map and the current set of point locations;

generating feedback associated with whether the first set of building codes is not satisfied; and

outputting the feedback to an end user of the mobile device.

3. The method of claim 2, further comprising:

determining a point in time when the first set of building codes will not be satisfied; and

alerting an end user of the mobile device if the point in time is within a particular time frame associated with a component lifetime of the first construction element.

4. The method of claim 3, wherein:

the determining a point in time when the first set of building codes will not be satisfied includes determining an average velocity of movement associated with the first construction element.

5. The method of claim 2, wherein:

the first construction element comprises a fire alarm; and

the first set of building codes is associated with fire prevention related regulations.

6. The method of claim 1, wherein:

the mobile device comprises a head-mounted display device; and

the one or more images comprise colored versions of the first construction element.

7. The method of claim 1, wherein:

the acquiring a 3D map of the structure includes generating the 3D map automatically without intervention by an end user of the mobile device; and

the updating a first virtual object is performed automatically without intervention by the end user of the mobile device.

8. The method of claim 1, wherein:

the 3D map of the structure only includes construction elements that have been locked.

9. One or more storage devices containing processor readable code for programming one or more processors to perform a method for verifying a structure using a mobile device comprising the steps of:

generating a 3D map of the structure as the structure is being constructed, the 3D map includes a virtual object associated with a first construction element of the structure, the virtual object is associated with a first set of point locations;

acquiring a first set of building codes associated with the structure;

determining that an end user of the mobile device is gazing at the first construction element;

determining a current set of point locations associated with the first construction element in response to the end user of the mobile device gazing at the first construction element;

determining whether the first set of building codes will not be satisfied at a future point in time based on the first set of point locations and the current set of point locations;

generating feedback associated with whether the first set of building codes will not be satisfied;

displaying on the mobile device one or more images associated with the feedback;

identifying a plurality of anchor points associated with the structure;

determining a plurality of locations associated with the plurality of anchor points; and

verifying a coordinate system associated with the 3D map based on the plurality of locations.

10. The one or more storage devices of claim 9, wherein:

the 3D map of the structure only includes construction elements that have been locked.

11. The one or more storage devices of claim 9, wherein:

the determining whether the first set of building codes will not be satisfied at a future point in time includes determining an average velocity of movement associated with the first construction element based on the first set of point locations and the current set of point locations.

12. The one or more storage devices of claim 9, wherein:

the first construction element comprises a fire alarm; and

the first set of building codes is associated with fire prevention related regulations.

13. The one or more storage devices of claim 9, wherein:

the mobile device comprises a head-mounted display device.

14. An electronic device for verifying a structure, comprising:

a memory, the memory stores a 3D map of the structure, the 3D map includes a virtual object associated with a first construction element of the structure, the memory stores a first set of building codes associated with the structure;

one or more processors, the one or more processors detect an end user of the electronic device gazing at the first construction element, the one or more processors determine a current set of point locations associated with the first construction element in response to the end user of the electronic device gazing at the first construction element, the one or more processors verify the first set of building codes based on the 3D map and the current set of point locations, the one or more processors generate feedback associated with whether the first set of building codes is not satisfied, the one or more processors render one or more images based on the feedback, the one or more processors identify a plurality of anchor points associated with the structure and verify a coordinate system associated with the 3D map based on the plurality of anchor points; and

a see-through display, the see-through display displays the one or more images.

15. The electronic device of claim 14, wherein:

the one or more processors determine a point in time when the first set of building codes will not be satisfied, the one or more processors determine the point in time based on an average velocity of movement associated with the first construction element.

16. The electronic device of claim 14, wherein:

the electronic device comprises a head-mounted display device.

17. The electronic device of claim 14, wherein:

the first construction element comprises a fire alarm; and

the first set of building codes is associated with fire prevention related regulations.