Systems and methods for presenting building information

• US 10,445,933 B2
• Filed: 03/30/2016
• Issued: 10/15/2019
• Est. Priority Date: 06/29/2011
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Rasane, Shailesh, Coleman, Ben, Sankarapandian, Rajesh, Janakiraman, Kirupakar, Chinnachamy, Parthasarathy, Sundar, Naga, Sasidharan, Sarath, Chen, Henry, Wang, Xiaoli, Beaulieu, Conrad Bruce, Raman, Thirumal, Mookan, Sekar, Gadde, Venu
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US15/085,805
• Publication Number:
  US 20160210790A1
• Time in Patent Office: 1,294 Days
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 50/10 Services
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Described herein are systems and methods for presenting building information. In overview, the technologies described herein provide relationships between Building Information Modeling (BIM) data (which includes building schematics defined in terms of standardized three dimensional models) and Building Management System (BMS) data (which includes data indicative of the operation of building components such as HVAC components and the like). Some embodiments use relationships between these forms of data thereby to assist technicians in identifying the physical location of particular pieces of equipment, for example in the context of performing inspections and/or maintenance. In some cases this includes the provision of 2D and/or 3D maps to portable devices, these maps including the location of equipment defined both in BIM and BMS data. In some cases, augmented reality technology is applied thereby to provide richer access to positional information.

Claims

1. A computer-implemented method for delivering to a user of a portable augmented reality display device data representative of a location of a piece of concealed equipment, the method comprising:

maintaining access to a 3D-object library that contains data representative of 3D models in the form of renderable objects, wherein each 3D renderable object is associated with a piece of equipment which is concealed behind a floor or a wall and is definable in Building Information Modelling (BIM) data in a BIM repository;

operating a 3D-object generator, the 3D-object generator being configured to correlate BIM data with 3D models present in the 3D-object library, thereby to generate a 3D model of the building by combining the 3D objects based on positional layouts defined by the BIM data in the BIM repository; and

operating a 3D transformer, which is configured to;

(i) identify a real-world marker in video image data from a video camera device, the real-world marker being at a known location defined relative to the BIM data in the BIM repository wherein the video camera device is provided on a device coupled to the augmented reality display device;

(ii) determine a real-time orientation of the real-world marker relative to the video camera device;

(iii) based on the determined real-time orientation of the real-world marker, apply a matrix transformation to a 3D image provided by the 3D-object generator to define an oriented 3D image such that the oriented 3D image is consistent with viewing the 3D object from a viewpoint of the video camera device;

(iv) rendering the oriented 3D image as an overlay on the video image data on a display of the portable device; and

repeating steps (ii) to (iv) for at least a subset of the video image data frames thereby to assist a user of the portable device in locating concealed equipment that is not able to be visually located.

2. A method according to claim 1 wherein the BIM repository comprises information about a building model in a common data format that defines parts of a structure/building as objects, which have an X-Y-Z geometry location.

3. A method according to claim 2 wherein the BIM data is presentable via a 3D graphic presentation with layers for a plurality of parts of the structure/building, including CAD design and one or more of:

HVAC, plumbing and electrical.

4. A method according to claim 3 wherein a plurality of entities defined in the BIM data are defined in a IFC data format, and wherein an IFC-BMS schema mapper is configured to map schema formats between the IFC data format and a predefined BMS application data format, thereby to enable association between a given IFC data format defined entity in the BIM data and a data set in BMS data representative of a piece of equipment and associated sensor points.

5. A method according to claim 4 including operating a BMS interface module, the BMS interface module being configured to convert and cause presentation of BMS data associated with a given IFC data format defined entity that is shown in the drawn 3D image.

6. A method according to claim 1 wherein the real-world marker is one of a plurality of real world markers mounted to walls of a building, thereby to enable positional identification of the camera device.

7. A method according to claim 1 wherein the augmented reality display device includes a smartphone.

8. A method according to claim 1 wherein the augmented reality display device includes head mounted video glasses.

9. A method according to claim 8 including identifying gesture driven inputs based on identification if movement of gesture driven markers mounted to the hands/fingers of a user.

10. A server, comprising:

a memory for storing to a 3D-object library and a Building information Modelling (BIM) repository, the 3D-object library including data representative of 3D models in the form of renderable objects, wherein each 3D renderable object is associated with a piece of equipment which is concealed behind a floor or a wall and is defined in Building Information Modelling (BIM) data in the BIM repository;

a 3D-object generator configured to correlate BIM data with 3D models present in the 3D-object library; and

a 3D transformer configured to;

(i) identify a real-world marker in video image data from a video camera device, the real-world marker being at a known location defined relative to the BIM data in the BIM repository, wherein the video camera device is provided on a device coupled to an augmented reality display device;

(ii) determine a real-time orientation of the real-world marker relative to the video camera device;

(iii) based on the determined real-time orientation of the real-world marker, apply a transformation to a 3D image provided by the 3D-object generator to define an oriented 3D image such that the oriented 3D image is consistent with viewing the 3D object from a viewpoint of the video camera device;

(iv) rendering the oriented 3D image as an overlay on the video image data on a display of the portable device; and

repeating steps (ii) to (iv) for at least a subset of the video image data frames thereby to assist a user of the portable device in locating concealed equipment that is not able to be visually located.

11. A server according to claim 10, wherein the BIM repository comprises information about a building model in a common data format that defines parts of a structure/building as objects, which have an X-Y-Z geometry location.

12. A server according to claim 11, wherein the BIM data is presentable via a 3D graphic presentation with layers for a plurality of parts of the structure/building, including CAD design and one or more of:

HVAC, plumbing and electrical.

13. A server according to claim 12, wherein a plurality of entities defined in the BIM data are defined in a IFC data format, and wherein an IFC-BMS schema mapper is configured to map schema formats between the IFC data format and a predefined BMS application data format, thereby to enable association between a given IFC data format defined entity in the BIM data and a data set in BMS data representative of a piece of equipment and associated sensor points.

14. A server according to claim 13, further comprising a BMS interface module that is configured to convert and cause presentation of BMS data associated with a given IFC data format defined entity that is shown in the drawn 3D image.

15. A server according to claim 10, wherein the real-world marker is one of a plurality of real world markers mounted to walls of a building, thereby to enable positional identification of the camera device.

16. A computer-implemented method for delivering to a user of a portable augmented reality display device data representative of a location of a piece of concealed equipment, the method comprising:

accessing a 3D-object library that contains data representative of 3D models in the form of renderable objects, wherein each 3D renderable object is associated with a piece of equipment which is concealed behind a floor or a wall and is definable in Building Information Modelling (BIM) data in a BIM repository;

operating a 3D-object generator, the 3D-object generator being configured to correlate BIM data with 3D models present in the 3D-object library, and to generate a 3D model of at least part of the building by combining the 3D objects based on positional layouts defined by the BIM data in the BIM repository; and

operating a 3D transformer configured to;

(i) identify a real-world marker in video image data from a video camera device, the real-world marker being at a known location defined relative to the BIM data in the BIM repository wherein the video camera device is provided on a device coupled to the augmented reality display device;

(ii) determine a real-time location and orientation of the real-world marker relative to the video camera device; and

(iii) based on the determined real-time location and orientation of the real-world marker, apply a transformation to a 3D image provided by the 3D-object generator to define an oriented 3D image such that the oriented 3D image is consistent with viewing the 3D object from a viewpoint of the video camera device;

(iv) rendering the oriented 3D image as an overlay on the video image data on a display of the portable device;

repeating steps (ii) to (iv) for at least a subset of the video image data frames thereby to assist a user of the portable device in locating concealed equipment that is not able to be visually located.

17. A method according to claim 16 wherein the BIM repository comprises information about a building model in a common data format that defines parts of a structure/building as objects, which have an X-Y-Z geometry location.

18. A method according to claim 17 wherein the BIM data is presentable via a 3D graphic presentation with layers for a plurality of parts of the structure/building, including CAD design and one or more of:

HVAC, plumbing and electrical.

19. A method according to claim 16 wherein the camera device comprises a smartphone.