Interactive method of presenting information in an image

• US 7,948,451 B2
• Filed: 06/17/2005
• Issued: 05/24/2011
• Est. Priority Date: 06/18/2004
• Status: Expired due to Fees

• Current Assignee: Forsvarets Materielverk, Totalforsvarets Forskningsinstitut (FOI)
• Original Assignee: Forsvarets Materielverk, Totalforsvarets Forskningsinstitut (FOI)
• Inventors: Persson, Bo, Sohlberg, Peter, Aldrin, Christina, Carleberg, Per, Gustafsson, Torbjörn
• Primary Examiner(s): Eisen; Alexander
• Assistant Examiner(s): PHAM, VIET DAVID
• Application Number:
  US11/629,868
• Publication Number:
  US 20080024392A1
• Time in Patent Office: 2,167 Days
• Field of Search: 345/7, 345/8, 345/679, 345/156, 345/633, 345/661, 345/676, 463 1- 8, 715/700
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

An interactive method of presenting information in an image to a user, who carries a head-mounted display device that presents an image of the surroundings. The user'"'"'s head position and direction are established in relation to the surroundings and a calculating device establishes, based on this, which part of the surroundings is positioned in front of the user'"'"'s head. Furthermore, the calculating device establishes where in a chain of information the user is positioned and, based on this, presents information in the form of a virtual image which is mixed with the real image. A central part of the image is then defined and fields and/or objects outside this area are shown in a non-disturbing way until the user'"'"'s gaze has left the central part. The user informs the calculating device that he/she has received the information and, respectively, performed the current operations by looking at the fields/objects.

Claims

1. An interactive method of presenting information in an image to a user within surroundings, using a system in which the user carries a head-mounted display device which presents an image of the surroundings to the user, said system operating in accordance with at least one chain of information having associated manipulations arranged in at least one chain of execution that is performed by a user in a hierarchical sequence of steps to complete a task, said method comprising the steps of:

establishing the user'"'"'s head position and direction in relation to the surroundings;

establishing, by a calculating device of said system based on the user'"'"'s head position and direction, which part of the surroundings is positioned in front of the user'"'"'s head as well as the user'"'"'s direction of gaze;

establishing, by the calculating device, where the user is in a chain of information about the part of the surroundings that is positioned in front of the user'"'"'s head, and where the user is in a chain of executed operations as regards the same part of the surroundings;

establishing, by the calculating device, on the basis of where the user is in the chain of information and where the user is in the chain of executed operations associated with said part of the surroundings, which information in the hierarchical sequence is next to be presented to the user and presenting the information in the form of a virtual image which is mixed with a real image as a mixed image that enables the user to perform a current operation as a next step within the hierarchical sequence for completing the task;

defining a central part of the mixed image, an area outside the central part including fields and/or objects that are shown in a way that is not disturbing to the user until the user'"'"'s direction of gaze has left said central part of the image and that are adapted to be included in a transfer of messages from the user to the calculating device, said transfer of messages enabling the user to move back and forth in the hierarchical sequence to obtain information needed to perform the current operation; and

the calculating device being informed that the user has received the information and, respectively, performed the current operation based on how the user'"'"'s direction of gaze is positioned in one or more of the fields and/or objects such that the user, by changing the user'"'"'s direction of gaze, initiates said transfer of messages to the calculating device;

recognizing, by said calculating device, said transfer of messages and establishing, in response to said transfer of messages, which information in said chain of information is next to be presented to the user to enable the user to perform a next step within the hierarchical sequence; and

presenting, by the calculating device, information relating to the next step in said hierarchical sequence making up said chain of executed operations for performance by said user to complete said task.

2. The method as claimed in claim 1, further comprising marking when the user'"'"'s direction of gaze in the mixed image is in a field or an object by said field or said object changing its appearance.

3. The method as claimed in claim 1, wherein the user'"'"'s direction of gaze is presented in a distinct manner when the user'"'"'s direction of gaze is positioned outside the central part of the image, said direction of gaze being either not presented or presented in a non-disturbing manner when said direction of gaze is in the central part of the image.

4. The method as claimed in claim 1, further comprising perceiving by the calculating device when the user has delivered a message by changing the user'"'"'s direction of gaze to a field or an object outside said central part, said field or said object serving as an input mechanism by which the user sends said message, said calculating device changing an appearance of said field or said object outside said central part to acknowledge that the message was received.

5. The method as claimed in claim 1, wherein the head-mounted display device mixes the real image and the virtual image using an “

optic-see-through”

technique.

6. The method as claimed in claim 1, wherein the head-mounted display device mixes the real image and the virtual image using a “

video-see-through”

technique.

7. The method as claimed in claim 2, wherein said change of appearance of said field or said object includes a change in color or shape.

8. The method as claimed in claim 3, wherein the user'"'"'s direction of gaze is presented as a cross when a point of fixation is outside said central part of the image.

9. The method as claimed in claim 4, wherein said change of appearance of said field or said object includes a change in color or shape.

10. An interactive method of presenting information in an image to a user who is positioned within surroundings, using a system in which the user carries a head-mounted display device which presents an image of the surroundings to the user, said system operating in accordance with at least one chain of information having associated manipulations arranged in at least one chain of execution that is performed by a user in a predefined hierarchical sequence of steps to complete a task, said method comprising the steps of:

establishing the user'"'"'s head position and direction in relation to the surroundings;

establishing, by a calculating device of said system, a part of the surroundings that is positioned in front of the user'"'"'s head and the user'"'"'s direction of gaze;

establishing, by the calculating device with respect to the part of the surroundings positioned in front of the user'"'"'s head, where the user is in a chain of information about said part of the surroundings and where the user is in a chain of operations to be executed that relate to said part of the surroundings;

determining, by the calculating device, which information in the chain of information is next to be presented to the user based on where the user is in the chain of information and where the user is in the chain of operations to be executed;

presenting the information to the user in the form of a virtual image which is mixed with a real image as a mixed image that enables the user to perform a current operation as a next step within the hierarchical sequence of step to complete the task;

defining a central part of the mixed image according to the user'"'"'s direction of gaze, an area outside the central part including fields and/or objects that are not shown in a way that would be disturbing to the user until the user'"'"'s direction of gaze has left said central part of the image; and

said calculating device being informed that the user has received the information and/or performed the operation from said chain of operations to be executed by said user sending an acknowledgment signal that is recognized by said calculating device.

11. The method as claimed in claim 10, wherein said user sends said acknowledgment signal by looking at a field or an object outside said central part, said field or said object serving as an input mechanism for the user to send said acknowledgment signal, and further comprising marking, by said system, when the calculating device has perceived a delivered acknowledgment signal by the field or the object outside said central part changing its appearance.

12. The method as claimed in claim 11, wherein said change of appearance of said field or said object includes a change in color or shape.

13. The method as claimed in claim 10, further comprising marking, by the system, when the user'"'"'s direction of gaze in the mixed image is in a field or an object by said field or said object changing its appearance.

14. The method as claimed in claim 13, wherein said change of appearance of said field or said object includes a change in color or shape.

15. The method as claimed in claim 10, wherein the user'"'"'s direction of gaze is presented in a distinct manner when the user'"'"'s direction of gaze is positioned outside the central part of the image, said direction of gaze being either not presented or presented in a non-disturbing manner when said direction of gaze is in the central part of the image.

16. The method as claimed in claim 10, wherein the head-mounted display device mixes the real image and the virtual image using an “

optic-see-through”

technique.

17. The method as claimed in claim 10, wherein the head-mounted display device mixes the real image and the virtual image using a “

video-see-through”

technique.

18. The method as claimed in claim 10, wherein said sending of the acknowledgment signal includes movement of the user'"'"'s direction of gaze relative to one or more fields and/or objects in a specific pattern or by gazing at a particular field or object for a specified time.

19. The method as claimed in claim 10, further comprising, after said calculating device has perceived a delivered acknowledgment signal, the step of said calculating device presenting a next step in the chain of information or chain of operations to be executed in response to said acknowledgment signal.