Augmented-reality system for situation-related support of the interaction between a user and an engineering apparatus

• US 8,373,618 B2
• Filed: 09/19/2007
• Issued: 02/12/2013
• Est. Priority Date: 03/02/1999
• Status: Active Grant

• Current Assignee: Siemens AG
• Original Assignee: Siemens AG
• Inventors: Friedrich, Wolfgang, Wohlgemuth, Wolfgang
• Primary Examiner(s): Zhou, Hong
• Application Number:
  US11/857,931
• Publication Number:
  US 20080100570A1
• Time in Patent Office: 1,973 Days
• Field of Search: 345 7- 9, 345/156, 345 11- 13, 345/629, 700/83, 700100-117, 359/13, 359/630
• US Class Current: 345/7
• CPC Class Codes: G05B 19/409 characterised by using manual data input [MDI] or by using control panel, e.g. controlling functions with the panel; characterised by control panel details or by setting parameters G05B19/408, G05B19/4093 take precedence
• CPC Class Codes: G05B 19/4183 characterised by data acquisition, e.g. workpiece identification
• CPC Class Codes: G05B 19/41875 characterised by quality surveillance of production
• CPC Class Codes: G05B 2219/31027 Computer assisted manual assembly CAA, display operation, tool, result
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 2219/35482 Eyephone, head-mounted 2-D or 3-D display, also voice and other control
• CPC Class Codes: G05B 2219/35494 Online documentation, manual, procedures, operator, user guidance, assistance
• CPC Class Codes: G05B 2219/35495 Messages to operator in multimedia, voice and image and text
• CPC Class Codes: Y02P 90/02 Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]

Abstract

The invention relates to an augmented-reality system having a mobile apparatus for the context-dependent insertion of assembly instructions. The context-dependent insertion of assembly instructions with process-optimized allocation of the necessary work steps provides situation-related support by for work sequences.

Claims

1. A situation-related support system for the interaction between a user and an apparatus comprising:

mobile equipment comprising;

a video camera for providing images of a current field of view of the user, including at least parts of the apparatus and a display, and

voice recording means for receiving voice data from the user,

a device for wireless communication coupled with said mobile equipment,

an augmented reality system comprising;

means for an analysis of video images which have been recorded by the video camera and have been wirelessly transmitted by the device for wireless communication to the augmented reality system,

means for automatically identifying one or more objects based on the analysis of the recorded video images;

means for automatically determining a particular situational context based on voice data received from the user via the voice recording means;

means for (a) automatically accessing documentation data related to the automatically identified one or more objects, wherein said documentation data related to the automatically identified one or more objects includes different data associated with different situational contexts, and (b) identifying from said documentation data the data associated with the particular situational context determined based on the voice data from the user, and

means for transmitting the context-relevant data to the device for wireless communication, wherein the display in the mobile equipment displays the context-relevant data.

2. The system according to claim 1, wherein the apparatus is an industrial installation.

3. The system according to claim 2, wherein the industrial installation comprises a further device for wireless communication and a communication means coupled with said augmented reality system.

4. The system according to claim 1, wherein the voice recording means comprises a microphone.

5. The system according to claim 1, wherein the display is located on lenses of data goggles, the video camera is an image-sensing apparatus located on the data goggles, and a microphone is located on the data goggles.

6. A situation-related support method for interaction between a user and an industrial installation, comprising the steps of:

recording video images of at least a part of said industrial installation by a mobile equipment comprising a video camera, voice recording means and a display, the at least a part of said industrial installation being within a field of view of the user;

transmitting voice data of the user from the mobile equipment to an automation system associated with said industrial installation;

wirelessly transmitting the video images from the mobile equipment to the automation system by a device,

transmitting said video images and voice data from the automation system associated with the industrial installation to an augmented reality system;

analyzing the video images in the augmented reality system,

automatically identifying one or more objects based on the analysis of the recorded video images,

automatically determining a particular situational context based on voice data received from the user via the voice recording means;

automatically accessing documentation data in the augmented reality system related to the automatically identified one or more objects, wherein said documentation data related to the automatically identified one or more objects includes different data associated with different situational contexts,

identifying from said documentation data the data associated with the determined particular situational context,

transmitting the context-relevant data, to the device for wireless communication,

displaying the context-relevant data.

7. The method according to claim 6, wherein the voice recording means comprises a microphone.

8. The method according to claim 6, wherein the display is arranged in data goggles and provides additional information in a field of”

view of a user and wherein the context-relevant data are overlayed in the field of view of the user.

9. The method according to claim 6, wherein the step of accessing documentation data comprises the step of accessing a remote data base or a remote expert.

10. A method for providing situational support to a user, comprising:

determining an industrial installation which requires service or repair;

recording voice data of the user,

recording video images of a portion of the industrial installation including multiple particular components;

wirelessly transmitting the video images and voice data to an augmented reality system;

analyzing the video images, identifying each of the multiple particular components from the video images, and accessing documentation data for each of said multiple particular components depending on the analysis;

determining from said documentation data and said voice data from said user a context-dependent instruction sequence, the context-dependent instruction sequence comprising instructions regarding each of the multiple particular components, and being pre-associated with a particular situational context determined based on the voice data from said user; and

displaying the context-dependent instruction sequence such that the user may view the context-dependent instructions substantially simultaneously with the component of the industrial installation, and such that the instructions regarding each of the multiple particular components are displayed in association with each of the multiple particular components simultaneously.

11. The method of claim 10, further comprising an intelligent tool operated by the user, wherein the method further comprises the step of receiving information from said intelligent tool and instructing the user how to operate said intelligent tool.

12. The method of claim 10, wherein the step of analyzing the video images comprises the step of determining at least one point of said industrial installation within an video image and displaying instructions in a field of view of the user relating to said one point.

13. The method of claim 12, wherein said instructions are displayed in a window and a graphical connection is drawn from said window to said one point.