Virtual control panel

• US 8,225,226 B2
• Filed: 12/20/2004
• Issued: 07/17/2012
• Est. Priority Date: 12/31/2003
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Research Ltd. (ABB Ltd.)
• Inventors: Skourup, Charlotte, Pretlove, John, Pettersen, Thomas
• Primary Examiner(s): Ke, Simon
• Assistant Examiner(s): TILLERY, RASHAWN N
• Application Number:
  US10/585,124
• Publication Number:
  US 20090300535A1
• Time in Patent Office: 2,766 Days
• Field of Search: 715/771, 715/773, 715/764, 715/856, 345/156, 700/259
• US Class Current: 715/771
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A system that enables a user to interact with a virtual control panel using a user controlled pointing object. The system includes a portable identification element, a tracking unit adapted to capture data representing the position of the pointing object and the position of the identification element, a storage unit, storing at least one pre-defined graphical interface representing a control panel of a device, a graphics unit, generating a graphical representation of the control panel, a registering unit, registering the graphical representation of the control panel in a fixed relation to the portable identification element to produce a virtual control panel and a display unit, showing the user a view including the real world and the virtual control panel.

Claims

1. A system that enables a user to interact with a virtual control panel for controlling an industrial device or system, the system comprising:

a pointing object carried and manipulated by the user for interacting with the virtual control panel to control the industrial device or system,

a first tracking unit adapted to capture data representing a position of the pointing object,

a portable identification element carried and manipulated by the user, the portable identification element being configured to define a position and orientation of the virtual control panel,

a second tracking unit adapted to capture data representing a position of the portable identification element,

a storage unit configured to store at least one pre-defined graphical interface representing a control panel of the industrial device or system,

a graphics unit configured to generate a graphical representation of the control panel based on said stored graphical interface,

a registering unit configured to register said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the portable identification element, to produce the virtual control panel,

a display unit configured to show the user a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

an application unit configured communicate with the industrial device or system to perform actions to control the industrial device or system in response to the interactions of the user with the virtual control panel, and configured to determine which actions to control the industrial device or system to perform based on the position of said user controlled pointing object in relation to the identification element.

2. The system according to claim 1, wherein the system is adapted to modify the appearance of the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

3. The system according to claim 1, wherein said graphical interface is adapted to display data from the device and wherein the system is adapted to generate a graphical representation of the data and to display the data on the virtual control panel.

4. The system according to claim 1, wherein said user controlled pointing object is a handheld pointing device or a part a body of the user.

5. The system according to claim 1, wherein the storage unit is adapted to store a plurality of graphical interfaces, each representing a control panel of a particular device, wherein the system is adapted to generate and display a plurality of graphical representations of control panels for different devices based on said stored graphical interfaces of the devices, and wherein the system further comprises:

an identification unit configured to identify which of the stored control panels to be displayed.

6. The system according to claim 5, wherein said identification unit comprises a recognition unit configured to recognize and identify devices in the environment of the user, and wherein the system is adapted to determine which of the stored control panels to be displayed based on which of the devices is identified.

7. The system according to claim 6, wherein said recognition unit is adapted to recognize and identify unique identification markings on the devices.

8. The system according to claim 5, wherein the system is arranged so the virtual control panel displayed changes when another device is recognized and identified, and when the user has accepted the other device.

9. The system according to claim 1, wherein said portable identification element is adapted to be carried by the user during interaction with the virtual control panel.

10. The system according to claim 1, wherein said portable identification element is attachable to a body of the user.

11. The system according to claim 1, wherein said display unit comprises a wearable display device showing the user said view.

12. A method that enables a user to interact with a virtual control panel for controlling an industrial device or system using a user controlled pointing object, the method comprising:

receiving data representing a position of the user controlled pointing object,

receiving data representing a position of a portable identification element carried and manipulated by the user,

storing at least one pre-defined graphical interface representing a control panel of the industrial device or system,

generating a graphical representation of the control panel of the device based on said pre-defined graphical interface,

registering said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the identification element, to produce the virtual control panel in a position and orientation defined by the user with the portable identification unit,

displaying a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

performing actions to communicate with the industrial device or system to control the industrial device or system in response to the interactions of the user with the virtual control panel, wherein the actions to control the industrial device or system to be performed is determined based on the position of said user controlled pointing object in relation to the position of the virtual control panel.

13. The method according to claim 12, further comprising:

modifying the appearance of the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

14. The method according to claim 12, further comprising:

defining a two-way communication between the virtual control panel and the device,

sending information to the device regarding the users actions with the virtual control panel,

receiving data from the device, generating a graphical representation of the received data and

displaying the data on the virtual control panel.

15. The method according to claim 14, wherein said data is displayed on the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

16. The method according to claim 12, further comprising:

storing a plurality of pre-defined graphical interfaces, each representing a control panel of a particular device,

determining which of the stored control panels to be displayed, and

generating a graphical representation of the control panel to be displayed based on the pre-defined graphical interface of the control panel to be displayed.

17. The method according to claim 12, wherein at least one of the stored graphical interfaces comprises more than one graphical view to be displayed on the virtual control panel, and which of the views to be displayed is determined based upon actions of the user.

18. The method according to claim 16, further comprising:

recognizing and identifying a device,

determining which of the stored control panels to be displayed based on the identified device, and

generating graphical representation of the control panel of the identified device based on the stored graphical interface of the identified device and displaying a view comprising the real world and the virtual control panel of the identified device projected in a fixed relation to said portable identification element.

19. The method according to claim 18, wherein each device is provided with a unique identification marking and a device is recognized by identifying its the unique identification marking.

20. The method according to claim 18, wherein the virtual control panel displayed is changed when another device is recognized and identified, and when the user has accepted the device.

21. The method according to claim 12, wherein said portable identification element is carried by the user during interaction with the virtual control panel.

22. The method according to claim 12, wherein the virtual control panel comprises virtual interaction members and an audio and/or visual feedback is generated when the user activates any of the virtual interaction members.

23. A computer program product, comprising:

a non-transitory computer readable medium; and

program instructions recorded on the computer readable medium which, when loaded into a computer, causes the computer to perform a method that enables a user to interact with a virtual control panel for controlling an industrial device or system using a user controlled pointing object, the method comprising

receiving data representing a position of the user controlled pointing object,

receiving data representing a position of a portable identification element carried and manipulated by the user,

storing at least one pre-defined graphical interface representing a control panel of the industrial device or system,

generating a graphical representation of the control panel of the device based on said pre-defined graphical interface,

registering said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the identification element, to produce the virtual control panel in a position and orientation defined by the user with the portable identification unit,

displaying a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

performing actions to communicate with the industrial device or system to control the industrial device or system in response to the interactions of the user with the virtual control panel, wherein the actions to control the industrial device or system to be performed is determined based on the position of said user controlled pointing object in relation to the position of the virtual control panel.