Virtual control panel

• US 20090300535A1
• Filed: 12/20/2004
• Published: 12/03/2009
• Est. Priority Date: 12/31/2003
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Research Ltd. (ABB Ltd.)
• Inventors: Skourup, Charlotte, Pretlove, John, Pettersen, Thomas
• Granted Patent: US 8,225,226 B2
• Time in Patent Office: Days
• US Class Current: 715/773
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

A system that enables a user to interact with a virtual control panel using a user controlled pointing object. The system includes a portable identification element, a tracking unit adapted to capture data representing the position of the pointing object and the position of the identification element, a storage unit, storing at least one pre-defined graphical interface representing a control panel of a device, a graphics unit, generating a graphical representation of the control panel, a registering unit, registering the graphical representation of the control panel in a fixed relation to the portable identification element to produce a virtual control panel and a display unit, showing the user a view including the real world and the virtual control panel.

Claims

1. A system that enables a user to interact with a virtual control panel using a user controlled pointing object, the system comprising:

a first tracking unit adapted to capture data representing the position of the pointing object,

a portable identification element,

a second tracking unit adapted to capture data representing the position of the identification element,

a storage unit storing at least one pre-defined graphical interface representing a control panel of a device, said graphical interface comprising an interface for user interactions with the device,

a graphics unit generating a graphical representation of the control panel based on said stored graphical interface,

a registering unit registering said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the identification element, to produce a virtual control panel,

a display unit showing the user a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

an application unit performing actions in response to the users interactions with the virtual control panel, and determining which actions to be performed based on the position of said user controlled pointing object in relation to the identification element.

2. The system according to claim 1, wherein the system is adapted to modify the appearance of the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

3. The system according to claim 1, wherein said graphical interface is adapted to display data from the device and wherein the system is adapted to generate a graphical representation of the data and to display the data on the virtual control panel.

4. The system according to claim 1, wherein said user controlled pointing object is a handheld pointing device or a part of the user'"'"'s body.

5. The system according to claim 1, wherein the storage unit is adapted to store a plurality of graphical interfaces, each representing a control panel of a particular device, wherein the system is adapted to generate and display a plurality of graphical representations of control panels for different devices based on said stored graphical interfaces of the devices, and wherein the system further comprises:

means for identifying which of the stored control panels to be displayed.

6. The system according to claim 5, wherein said means for identifying which of the stored control panels to be displayed comprises a recognition unit for recognizing and identifying devices in the environment of the user, and wherein the system is adapted to determine which of the stored control panels to be displayed based on which of the devices is identified.

7. The system according to claim 6, wherein said recognition unit is adapted to recognize and identify unique identification markings on the devices.

8. The system according to claim 5, wherein the system is arranged so that it changes the virtual control panel displayed when another device is recognized and identified, and when the user has accepted the device.

9. The system according to claim 1, wherein said portable identification element is adapted to be carried by the user during interaction with the virtual control panel.

10. The system according to claim 1, wherein said portable identification element is attachable to the body of the user.

11. The system according to claim 1, wherein said display unit comprises a wearable display device showing the user said view.

12. A method that enables a user to interact with a virtual control panel using a user controlled pointing object, the method comprising:

receiving data representing the position of the user controlled pointing object,

receiving data representing the position of a portable identification element,

storing at least one pre-defined graphical interface representing a control panel of a device,

generating a graphical representation of the control panel of the device based on said pre-defined graphical interface,

registering said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the identification element, to produce a virtual control panel,

displaying a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

performing actions in response to the users interactions with the virtual control panel, wherein the actions to be performed is determined based on the position of said user controlled pointing object in relation to the position of the virtual control panel.

13. The method according to claim 12, further comprising:

modifying the appearance of the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

14. The method according to claim 12, further comprising:

defining a two-way communication between the virtual control panel and the device,

sending information to the device regarding the users actions with the virtual control panel,

receiving data from the device, generating a graphical representation of the received data and

displaying the data on the virtual control panel.

15. The method according to claim 14, wherein said data is displayed on the virtual control panel in response to interactions between the user controlled pointing object and the virtual control panel.

16. The method according to claim 12, further comprising:

storing a plurality of pre-defined graphical interfaces, each representing a control panel of a particular device,

determining which of the stored control panels to be displayed, and

generating a graphical representation of the control panel to be displayed based on the pre-defined graphical interface of the control panel to be displayed.

17. The method according to claim 12, wherein at least one of the stored graphical interfaces comprises more than one graphical view to be displayed on the virtual control panel, and which of the views to be displayed is determined based upon the users actions.

18. The method according to claim 16, further comprising:

recognizing and identifying a device,

determining which of the stored control panels to be displayed based on the identified device, and

generating graphical representation of the control panel of the identified device based on the stored graphical interface of the identified device and displaying a view comprising the real world and the virtual control panel of the identified device projected in a fixed relation to said portable identification element.

19. The method according to claim 18, wherein each device is provided with a unique identification marking and a device is recognized by identifying its unique identification marking.

20. The method according to claim 18, wherein the virtual control panel displayed is changed when another device is recognized and identified, and when the user has accepted the device.

21. The method according to claim 12, wherein said portable identification element is carried by the user during interaction with the virtual control panel.

22. The method according to claim 12, wherein the virtual control panel comprises virtual interaction members and an audio and/or visual feedback is generated when the user activates any of the virtual interaction members.

23. A computer program product, comprising:

a computer readable medium; and

program instructions recorded on the computer readable medium which, when loaded into a computer, causes the computer to perform the steps of

receiving data representing the position of the user controlled pointing object,

receiving data representing the position of a portable identification element,

storing at least one pre-defined graphical interface representing a control panel of a device,

generating a graphical representation of the control panel of the device based on said pre-defined graphical interface,

registering said graphical representation of the control panel in a fixed relation to said portable identification element, based on said data representing the position of the identification element, to produce a virtual control panel,

displaying a view comprising the real world and the virtual control panel projected in a fixed relation to said portable identification element, and

performing actions in response to the users interactions with the virtual control panel, wherein the actions to be performed is determined based on the position of said user controlled pointing object in relation to the position of the virtual control panel.

24. (canceled)