Augmented reality system and method

• US 20040189675A1
• Filed: 12/23/2003
• Published: 09/30/2004
• Est. Priority Date: 12/30/2002
• Status: Active Grant

• Current Assignee: ABB Schweiz AG (ABB Limited)
• Original Assignee: ABB Research Ltd. (ABB Limited)
• Inventors: Pretlove, John, Pettersen, Thomas
• Granted Patent: US 7,714,895 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: B25J 9/1656 characterised by programming, planning systems for manipulators
• CPC Class Codes: G05B 19/427 Teaching successive positions by tracking the position of a joystick or handle to control the positioning servo of the tool head, master-slave control G05B19/423 takes precedence
• CPC Class Codes: G05B 2219/35482 Eyephone, head-mounted 2-D or 3-D display, also voice and other control
• CPC Class Codes: G05B 2219/36452 Touch points with handheld probe, camera detects position and orientation probe
• CPC Class Codes: G05B 2219/39449 Pendant, pda displaying camera images overlayed with graphics, augmented reality
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

An augmented reality system comprising: a camera (19) for capturing an image, the camera being movably locatee at a local site, a registering unit (9), generating graphics and registering the generated graphics to the image from the camera, to provide a composite augmented reality image, a display device (5) located at a remote site, physically separated from the local site, for displaying a view comprising the composite augmented reality image, and a communication link (1), for communication of information between the local and the remote site, and a specifying unit (7), for specification of a position and an orientation in the remote site. The registering unit is adapted for registering the generated graphical representation to the image in dependence of the specified position and orientation, and the camera is arranged such that its position and orientation is dependent on the specified position and orientation.

Claims

1. An augmented reality system comprising:

a camera for capturing an image, the camera being movably located at a local site,

a registering unit, generating graphics and registering the generated graphics to the image from the camera, to provide a composite augmented reality image,

a display device located at a remote site, physically separated from the local site, for displaying a view comprising the composite augmented reality image, and

a communication link, for communication of information between the local and the remote site,

wherein the system further comprises a specifying unit, for specification of a position and an orientation in the remote site, the camera is arranged such that its position and orientation is dependent on the specified position and orientation, and the registering unit is adapted for registering the generated graphics to the image in dependence of the specified position and orientation.

2. The system according to claim 1, wherein said specifying unit comprises a tracking unit, adapted for determining the position and orientation of a movable device located at the remote site, the registering unit is adapted for registering the generated graphics to the image in dependence of the position and orientation of the movable device, and the camera is arranged such that its position and orientation are dependent on the position and orientation of the movable device.

3. The system according to claim 2, wherein said movable device is the display device.

4. The system according to claim 1, further comprising a robot located at the local site, the camera is mounted on the robot and the robot is arranged in such a manner that the movement of the robot depends on the specified position and orientation.

5. The system according to claim 1, further comprising a graphical generator, for generation of a graphical representation, and the registering unit is adapted for generating graphics based on the graphical representation.

6. The system according to claim 1, further comprising operator input means, located at the remote site, provided for feeding data related to the graphics to be displayed to the system, and the system is adapted for generating the graphics based on said data.

7. The system according to claim 6, wherein said operator input means comprises a pointing device and a tracking unit for determining the position of the pointing device and that the system is adapted for generating a graphical representation of a point being presently pointed out by the pointing member based on the position of the pointing device.

8. The system according to claim 1, further comprising a second specifying unit, for specifying a position and an orientation in the local site, a second registering unit, generating graphics and registering the generated graphics to the real environment or an image of the environment of the local site, in dependence of the position and orientation specified by the second specifying unit, and a local display device adapted for displaying a view comprising the environment of the local site and said generated graphics projected on the environment.

9. The system according to claim 8, further comprising a second movable device located at the local site, the second specifying unit comprises a second tracking unit, for determining the position and the orientation of the second movable device.

10. The system according to claim 11, wherein said second movable device is the local display device.

11. The system according claim 9, further comprising a second camera for capturing an image, the camera being arranged in a fix relation to the second movable device, and the second registering unit, is adapted for registering the generated graphics to the image from the second camera, to provide a composite augmented reality image, and that the local display device is adapted for displaying a view comprising the composite augmented reality image.

12. A The system according to claim 8, wherein the remote display device is adapted for displaying a view seen from a first visual angle that depends on the position and orientation received from the first mentioned specifying unit and the local display device is adapted for displaying the same view as the remote display device seen from a second visual angle that depends on the position and orientation received from the second specifying unit.

13. The system according to claim 1, further comprising means for transferring voices between the remote and the local site via the communication link.

14. The system according to claim 1, wherein the communication link is a network.

30. Use of a system according to claim 1 for remote programming of an industrial robot.

31. The system according to claim 11, further comprising a handheld display device comprising the display member and the camera.

32. The system according to claim 31, wherein the handheld display device is arranged so that the user seems to look directly through the display.

33. Use of the method according to claim 1 for a paint application.

15. A method for remotely displaying an augmented reality view comprising graphical information overlaid an image captured at a local site, the method comprising:

specifying a position and an orientation at a remote site that is physically separated from the local site,

positioning and orientating a camera, located at the local site, according to the specified position and orientation,

obtaining an image from the camera,

generating graphics,

generating a composite augmented reality image based on the image, the graphics, and the specified position and orientation, and

displaying a view comprising the composite augmented reality image.

16. The method according to claim 15, wherein specifying a position and an orientation comprises determining the position and orientation of a movable device located at the remote site and the camera is positioned and orientated according to the position and orientation of the movable device.

17. The method according to claim 16, wherein said movable device is a remote display device and that said view comprising the composite augmented reality image is displayed on the remote display device.

18. The method according to claim 15, further comprising controlling the movements of a robot, having the camera mounted thereon, according to the position and orientation of the movable device.

19. The method according to claim 15, further comprising obtaining data related to the graphics to be displayed, and generating the graphics based on said data.

20. The method according to claim 15, further comprising receiving information about the position of a pointing device and generating graphics representing a point, being presently pointed out by the pointing member, based on the position of the pointing device.

21. The method according to claim 15, further comprising specifying a position and an orientation in the local site, and displaying a second view comprising the environment of the local site and the generated graphics projected on the environment in dependence of the locally specified position and orientation.

22. The method according to claim 21, wherein specifying a position and an orientation in the local site comprises determining the position and orientation of a second movable device located at the local site.

23. The method according to claim 22, wherein the second movable device is a local display device and that said second view, comprising the environment of the local site and the graphics, is displayed on the local display device.

24. The method according to claim 22, further comprising capturing an image from a second camera being arranged in a fix relation to the second movable device, and registering the generated graphics to the image from the second camera, to provide a composite augmented reality image, and displaying a view comprising the composite augmented reality image on the local display device.

25. The method according to claim 21, further comprising generating second graphics and displaying the second view comprising the environment of the local site and the second graphics projected on the environment in dependence of the specified position and orientation.

26. The method according to claim 25, further comprising generating a local graphical representation, generating a remote graphical representation, transferring the local and remote graphical representations between the local and the remote site, generating the first mentioned graphics based on the local and the remote graphical representation, and generating the second graphics based on the local and the remote graphical representation.

27. The method according to claim 21, wherein the view displayed in the remote site comprises the environment of the local site and the overlaid graphics seen from an visual angle that depends on the position and orientation specified in the remote site and the view displayed in the local site comprises the environment of the local site and the overlaid graphics seen from an visual angle that depends on the position and orientation specified in the local site.

28. A computer program product directly loadable into the internal memory of a computer, comprising software code portions for performing the steps of claim 15, when said product is run on a computer.

29. A computer readable medium having a program recorded thereon, where the program is to make a computer perform the steps of claim 15, when said program is run on the computer.