Virtual work place

• US 8,046,719 B2
• Filed: 05/30/2007
• Issued: 10/25/2011
• Est. Priority Date: 05/31/2006
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Technology Limited (ABB Ltd.)
• Inventors: Loekstad, Trond, Engedal, Torbjørn, Skourup, Charlotte, Pretlove, John
• Primary Examiner(s): BONSHOCK, DENNIS G
• Application Number:
  US11/806,208
• Publication Number:
  US 20080005702A1
• Time in Patent Office: 1,609 Days
• Field of Search: 715/848, 715/850, 715/853, 715/863, 715/755, 715/757, 715/782
• US Class Current: 715/848
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00

Abstract

The present invention is concerned with a method and computer-based system for configuring, monitoring and/or operating a graphical user interface in two or three dimensions. In particular it is concerned with a graphical user interface used for controlling, monitoring and/or operating equipment in a control system of an industrial process or an installation for oil and gas or for an installation for electrical power generating, transmission and/or distribution.

Claims

1. A system, comprising:

a logic unit for managing data in a plurality of applications;

a wearable display device for visually displaying the data received from the logic unit whereby a user can receive data from the wearable display;

an augmented reality representation of computer-generated graphical information overlaid on an image of a real world;

an interactive input device;

a tracking system to capture data representing the position of the wearable display device, relative to the interactive input device;

means for displaying an augmented reality representation of a virtual desktop on said wearable display device, the virtual desktop comprising a plurality of functional interfaces representing said applications, wherein the interfaces comprise at least one of technical information and control functions, accessed using said interactive input device, wherein said virtual desktop is arranged expanded in a 3D space so that the virtual desktop is shaped as a virtual sphere surrounding the user and configured to display the functional interfaces at an optional depth in the sphere, such that the interfaces are simultaneously displayed, and wherein the interactive input device is configured to interact with the virtual desktop to manipulate the interfaces in the 3D space, wherein the expansion of said virtual desktop is 360 degrees in three degrees of freedom, and wherein the manipulation of the functional interfaces comprises at least one of navigating among, movement, rotation, scaling, or locking in the 3D space coordinated using polar or Cartesian coordinates, minimizing, zooming in, zooming out, selecting data, entering data, copying data, pasting data, or changing data; and

a camera to capture a live video stream of the environment corresponding to a view of the real world of the user, wherein the virtual graphics are overlaid on the video of said real world forming said augmented reality representation of a virtual desktop.

2. The system according to claim 1, wherein the interactive input device is used to navigate among, move, rotate, scale and lock the functional interfaces in the 3D space.

3. The system according to claim 1, wherein the interactive input device is configured to interact with an application accessible via one of the functional interfaces.

4. The system according to claim 1, wherein the system is adapted to two or more users working together on one application simultaneously.

5. The system according to claim 1, wherein the display device is adapted to be optically transparent and the computer-generated graphics are registered directly onto the display device and follows a view of the user of the real world so that the computer-generated graphics are overlaid the real world forming the augmented reality representation of a virtual desktop.

6. The system according to claim 1, wherein the wearable display device comprises at least one of instance glasses, a head-mounted display, a head-up display, or virtual retinal display.

7. The system according to claim 1, wherein the interactive input device comprises at least one of keyboard, 2D-mouse, 3D-mouse, joystick, trackball, data gloves, or means to interact with a pen/fingertip directly on the functional interfaces functioning as a virtual touch screen.

8. The system according to claim 1, wherein two or more users share views with each other.

9. The system according to claim 1, wherein two or more users interact with at least one functional interface simultaneously.

10. The system according to claim 1, wherein two or more users interact with a same virtual desktop.

11. A system comprising a computer including computer program instructions for generating a graphical user interface, a wearable display device for visually displaying the graphical user interface, a tracking system to capture data representing the position of the wearable display device, relative to the interactive input device, a camera to capture real world video of an environment corresponding to a view of a real world of a user, the graphical user interface comprising

a virtual desktop comprising a plurality of simultaneously displayed functional interfaces representing applications, wherein the interfaces comprise at least one of technical information and control functions, accessed using an interactive input device, wherein the interactive input device is configured to interact with the virtual desktop to manipulate the interfaces in the 3D space, wherein said virtual desktop is arranged expanded in a 3D space so that the virtual desktop is shaped as a virtual sphere surrounding a user and with the functional interfaces simultaneously displayed at an optional depth in the sphere, wherein the virtual desktop overlaid on the real world video, thereby forming said augmented reality representation of a virtual desktop, and wherein an arrangement of the interfaces in the 3D space is configured to be altered by interactions between the interactive input device and the virtual desktop, wherein the virtual desktop is expandable 360 degrees in three degrees of freedom, and wherein the functional interfaces are manipulatable through at least one of navigating among, movement, rotation, scaling, or locking in the 3D space coordinated using polar or Cartesian coordinates, minimizing, zooming in, zooming out, selecting data, entering data, copying data, pasting data, or changing data.

12. A method that enables a user to interact with a system comprising a logic unit for managing data in a plurality of applications using a wearable display device, for visually displaying the data received from the logic unit including means for displaying the data in a forward field of view of the user, a camera to capture a live video stream of the environment corresponding to a view of the real world of the user, and a receiver configured to receive data from the wearable display device, the method comprising:

displaying an augmented reality representation of computer-generated graphical information overlaid on an image of a real world;

using an interactive input device, a tracking system to capture data representing the position of the wearable display device, relative to the interactive input device;

displaying an augmented reality representation of a virtual desktop on said wearable display device, said virtual desktop comprising applications represented by a plurality of simultaneously displayed functional interfaces comprising at least one of technical information and control functions, wherein the virtual graphics are overlaid on the video of said real world from the camera, thereby forming said augmented reality representation of a virtual desktop;

performing actions so that said interactive input device accesses said functional interfaces, wherein the interactive input device interacts with the virtual desktop to manipulate the interfaces in the 3D space;

performing actions so that said desktop is arranged expanded in a 3D space so that the desktop is shaped as a virtual sphere surrounding the user and with the functional interfaces simultaneously displayed at optional depth in the sphere; and

expanding said virtual desktop in 360 degrees in three degrees of freedom, wherein manipulating the interfaces comprises at least one of navigating among, moving the functional interfaces, rotating, scaling or locking the functional interfaces in 3D space using Polar or Cartesian coordinates to determine the locations of the functional interfaces in space, minimizing, zooming in, zooming out, selecting data, entering data, copying data, pasting data, or changing data.

13. The method according to claim 12, further comprising:

modifying the appearance of virtual desktop in response to interactions between the controlled interactive input device and the virtual desktop.

14. The method according to claim 13, wherein manipulating the interfaces comprises using the interactive input device to carry out at least one of navigate among, move, rotate, scale or lock functional interfaces in the 3D space, minimize, zoom in, zoom out, select data, enter data, copy data, paste data, or change data.

15. The method according to claim 13, wherein the interactive input device is used to interact with an application accessible via a functional interface.

16. The method according to claim 12, wherein two or more users are working together on one application simultaneously.

17. The method according to claim 12, further comprising:

using a camera to capture a live video stream of the environment corresponding to the user'"'"'s view of the real world and that the virtual graphics are overlaid on the video of said real world.

18. The method according to claim 12, further comprising:

using a display device that is adapted to be optically transparent and registering the computer-generated graphics directly onto the display device following a view of the user of the real world so that the computer-generated graphics are overlaid the real world forming.

19. The method according to claim 12, wherein the wearable display device comprises at least one of instance glasses, a head-mounted display, a head-up display, and virtual retinal display.

20. The method according to claim 12, wherein the interactive input device comprises at least one of keyboard, 2D-mouse, 3D-mouse, joystick, trackball, data gloves or means to interact with a pen/fingertip directly on the functional interfaces functioning as a virtual touch screen.

21. The method according to claim 12, further comprising:

controlling a system of an industrial process or an installation for oil and gas or for an installation for electrical power generating, transmission and/or distribution.

22. A computer program product, comprising:

a non-transitory computer readable medium; and

computer program instructions recorded on the computer readable medium and executable by a processor for carrying our a method the method comprising displaying an augmented reality representation of computer-generated graphical information overlaid on an image of a real world, using an interactive input device, a tracking system to capture data representing the position of the wearable display device, relative to the interactive input device, displaying an augmented reality representation of a virtual desktop on said wearable display device, said virtual desktop comprising applications represented by a plurality of simultaneously displayed functional interfaces comprising at least one of technical information and control functions, wherein the virtual graphics are overlaid on the video of said real world from the camera, thereby forming said augmented reality representation of a virtual desktop, performing actions so that said interactive input device accesses said functional interfaces, wherein the interactive input device interacts with the virtual desktop to manipulate the interfaces in the 3D space, performing actions so that said desktop is arranged expanded in a 3D space so that the desktop is shaped as a virtual sphere surrounding the user and with the functional interfaces simultaneously displayed at optional depth in the sphere; and

expanding said virtual desktop in 360 degrees in three degrees of freedom, wherein manipulating the interfaces comprises at least one of navigating among, moving the functional interfaces, rotating, scaling or locking the functional interfaces in 3D space using Polar or Cartesian coordinates to determine the locations of the functional interfaces in space, minimizing, zooming in, zooming out, selecting data, entering data, copying data, pasting data, or changing data.