Augmented reality building operations tool

• US 8,830,267 B2
• Filed: 11/15/2010
• Issued: 09/09/2014
• Est. Priority Date: 11/16/2009
• Status: Expired due to Fees

• Current Assignee: Alliance For Sustainable Energy LLC
• Original Assignee: Alliance For Sustainable Energy LLC
• Inventors: Brackney, Larry J.
• Primary Examiner(s): Thompson, James A
• Application Number:
  US12/946,455
• Publication Number:
  US 20110115816A1
• Time in Patent Office: 1,394 Days
• Field of Search: None
• US Class Current: 345/633
• CPC Class Codes: F24F 11/56 Remote control
• CPC Class Codes: F24F 11/64 using pre-stored data
• CPC Class Codes: F24F 11/65 for selecting an operating mode
• CPC Class Codes: F24F 2110/10 Temperature
• CPC Class Codes: F24F 2110/20 Humidity
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2380/00 Specific applications
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H05B 47/10 Controlling the light source
• CPC Class Codes: H05B 47/19 via wireless transmission
• CPC Class Codes: H05B 47/1965 using handheld communication devices

Abstract

A method (700) for providing an augmented reality operations tool to a mobile client (642) positioned in a building (604). The method (700) includes, with a server (660), receiving (720) from the client (642) an augmented reality request for building system equipment (612) managed by an energy management system (EMS) (620). The method (700) includes transmitting (740) a data request for the equipment (612) to the EMS (620) and receiving (750) building management data (634) for the equipment (612). The method (700) includes generating (760) an overlay (656) with an object created based on the building management data (634), which may be sensor data, diagnostic procedures, or the like. The overlay (656) is configured for concurrent display on a display screen (652) of the client (642) with a real-time image of the building equipment (612). The method (700) includes transmitting (770) the overlay (656) to the client (642).

Claims

1. A method, comprising:

with an augmented reality building operations tool (ARBOT) server, receiving from a mobile client in a building an augmented reality request for equipment of a building system that is managed by an energy management system (EMS);

in response, receiving, at the ARBOT server from the EMS, building management data for the equipment of the building system;

with the ARBOT server, generating an overlay with at least one object based on the building management data, wherein the overlay is configured for concurrent display on a display screen of the mobile client with a real-time image of the equipment in the building;

transmitting the overlay to the mobile client for display with the real-time image of the equipment,

wherein a display location of the at least one object in the overlay is aligned with a position of the equipment in the real-time image such that the at least one object is displayed in the display screen proximate to or at least partially overlaying the real-time image of the equipment;

after the transmitting of the overlay to the mobile client, receiving a client interaction request from the mobile client including a user selection of the at least one object in the overlay; and

operating the ARBOT server to obtain additional building management data for the equipment associated with the at least one object in the overlay and to provide the additional building management data to the mobile client for display on the display screen.

2. The method of claim 1, wherein the augmented reality request includes an identification of the equipment and wherein the building management data includes collected data from a sensor in the building operating to monitor operation of the identified equipment.

3. The method of claim 2, wherein the identification of the equipment is determined by processing a fiducial marker in the real-time image or based upon a determined position of the mobile client.

4. The method of claim 1, wherein the additional building management data includes an actuator-in-hand associated with the equipment.

5. The method of claim 1, wherein the additional building management data is presented to the mobile client as an additional overlay for display in the display screen.

6. The method of claim 1, wherein the overlay includes a selectable icon requesting operator input and wherein the ARBOT server responds to selection of the selectable icon by storing in memory an action item associated with the equipment.

7. A building management system for use with an energy management system (EMS) storing building management data associated with building system components in a building, comprising:

a server with a processor running ARBOT software and communicatively linked to the EMS; and

wherein a client, positioned in the building and wirelessly communicating with the server, displays an image of one of the components on a display screen,

wherein the server accesses the EMS to retrieve building management data associated with the displayed component, the server generates an overlay for the displayed image that includes an object based on the retrieved building management data and transmits the overlay to the client,

wherein the client updates the display screen to concurrently display the image of the component and the overlay, with the overlay being displayed to be proximate to or overlaying the image of the component on the display screen,

wherein the server stores use case data for a plurality of the components including diagnostic or maintenance processes, wherein the server processes data received from the client to identify the component, and wherein the overlay includes at least a portion of the use case data for the identified component,

wherein the overlay includes an interactive element receiving input from an operator of the client requesting additional information from the EMS, and

wherein the server retrieves the additional information from the EMS and creates an additional overlay for the display by the client over the image based on the additional information.

8. The system of claim 7, wherein the building management data includes operating data for the component collected by a sensor in the building.

9. The system of claim 7, wherein the building management data includes diagnostic or maintenance information for the component that is provided in the object.

10. The system of claim 7, wherein a display location of the object is aligned in the image to a location in the image of the component and includes a portion indicating an operating state of the component with a graphical element.