AUGMENTED REALITY BUILDING OPERATIONS TOOL

• US 20110115816A1
• Filed: 11/15/2010
• Published: 05/19/2011
• Est. Priority Date: 11/16/2009
• Status: Active Grant

• Current Assignee: Alliance For Sustainable Energy LLC
• Original Assignee: Alliance For Sustainable Energy LLC
• Inventors: Brackney, Larry J.
• Granted Patent: US 8,830,267 B2
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: F24F 11/56 Remote control
• CPC Class Codes: F24F 11/64 using pre-stored data
• CPC Class Codes: F24F 11/65 for selecting an operating mode
• CPC Class Codes: F24F 2110/10 Temperature
• CPC Class Codes: F24F 2110/20 Humidity
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H05B 47/10 Controlling the light source
• CPC Class Codes: H05B 47/19 via wireless transmission
• CPC Class Codes: H05B 47/1965 using handheld communication devices

Abstract

A method (700) for providing an augmented reality operations tool to a mobile client (642) positioned in a building (604). The method (700) includes, with a server (660), receiving (720) from the client (642) an augmented reality request for building system equipment (612) managed by an energy management system (EMS) (620). The method (700) includes transmitting (740) a data request for the equipment (612) to the EMS (620) and receiving (750) building management data (634) for the equipment (612). The method (700) includes generating (760) an overlay (656) with an object created based on the building management data (634), which may be sensor data, diagnostic procedures, or the like. The overlay (656) is configured for concurrent display on a display screen (652) of the client (642) with a real-time image of the building equipment (612). The method (700) includes transmitting (770) the overlay (656) to the client (642) for display with the real-time image of the equipment (612). The equipment (612) is identified using a client position or a fiduciary marker (614) on or near the equipment (612).

Claims

1. A method, comprising:

with an augmented reality building operations tool (ARBOT) server, receiving from a mobile client an augmented reality request for equipment of a building system that is managed by an energy management system (EMS);

in response, receiving at the ARBOT server from the EMS building management data for the equipment of the building system;

with the ARBOT server, generating an overlay with at least one object based on the building management data, wherein the overlay is configured for concurrent display on a display screen of the mobile client with a real-time image of the equipment in the building; and

transmitting the overlay to the mobile client for display with the real-time image of the equipment.

2. The method of claim 1, wherein the data request includes an identification of the equipment and wherein the building management data includes collected data from a sensor in the building operating to monitor operation of the identified equipment.

3. The method of claim 2, wherein the identification of the equipment is determined by processing a fiducial marker in the real-time image or based upon a determined position of the mobile client.

4. The method of claim 3, wherein the overlay object is mapped to the position of the equipment in the real-time image.

5. The method of claim 1, after the transmitting of the overlay to the mobile client, receiving a client interaction request from the mobile client based on a selection of a portion of the overlay and, in response, operating the ARBOT server to obtain additional building management data for the equipment and to provide the additional building management data to the mobile client for display on the display screen.

6. The method of claim 5, wherein the additional building management data includes an actuator-in-hand associated with the equipment.

7. The method of claim 5, wherein the additional building management data is presented to the mobile client as an additional overlay for display in the display screen.

8. The method of claim 1, wherein the overlay includes a selectable icon requesting operator input and wherein the ARBOT server responds to selection of the selectable icon by storing in memory an action item associated with the equipment.

9. A building management system for use with an energy management system (EMS) storing building management data associated with building system components in a building, comprising:

a server with a processor running ARBOT software and communicatively linked to the EMS; and

wherein a client positioned in the building and wirelessly communicating with the server, displays an image of one of the components on a display screen,

wherein the server accesses the EMS to retrieve building management data associated with the displayed component, the server generates an overlay for the displayed image that includes an object based on the retrieved building management data and transmits the overlay to the client, and

wherein the client updates the display screen to concurrently display the image of the component and the overlay.

10. The system of claim 9, wherein the building management data includes operating data for the component collected by a sensor in the building.

11. The system of claim 9, wherein the building management data includes diagnostic or maintenance information for the component that is provided in the object.

12. The system of claim 9, wherein the overlay includes an interactive element receiving input from an operator of the client requesting additional information from the EMS and wherein the server retrieves the additional information from the EMS and creates an additional overlay for the display by the client over the image based on the additional information.

13. The system of claim 9, wherein the object is mapped in the image to a location in the image of the component and includes a portion indicating an operating state of the component with a graphical element.

14. The system of claim 9, wherein the server stores use case data for a plurality of the components including diagnostic or maintenance processes, wherein the server processes data received from the client to identify the component, and wherein the overlay includes at least a portion of the use case data for the identified component.

15. A computer-readable storage medium with an executable program stored thereon, wherein the program is executable to instruct a processor to perform the following:

receive a message from a client with digital data identifying a component of building system;

retrieve, from an EMS associated with the building system, building management data for the component;

generate an overlay including a building management object including at least a portion of the building management data; and

transmit the overlay to the client over a wireless connection for display with a real-time image of the component.

16. The computer-readable storage medium of claim 15, wherein the building management object is mapped to a location of the component in the real-time image.

17. The computer-readable storage medium of claim 15, wherein the portion of the building management data included in the building management object includes operational data for the component collected by one or more sensors in the building and further includes an indicator indicating a present operating status of the component relative to an operating standard defined for the component.

18. The computer-readable storage medium of claim 15, wherein the overlay includes an interactive portion configured to receive user input requesting additional information regarding the component or requesting storage in memory an action item for the component.

19. The computer-readable storage medium of claim 15, wherein the program is further executable to instruct the processor to determine the identification of the component by processing the digital data identifying the component received from the client.

20. The computer-readable storage medium of claim 15, wherein retrieving the building management data for the component is performed based on a predefined use case stored in memory for the component that defines a subset of the building management data to retrieve for the component and to present in the overlay.