Automatic text scrolling on a display device

• US 8,767,014 B2
• Filed: 07/28/2011
• Issued: 07/01/2014
• Est. Priority Date: 07/22/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Vaught, Benjamin I., Kipman, Alex Aben-Athar, Crocco, Robert
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US13/193,563
• Publication Number:
  US 20130021373A1
• Time in Patent Office: 1,069 Days
• Field of Search: 345/633, 345/8
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2380/02 Flexible displays

Abstract

A see-through head-mounted display (HMD) device, e.g., in the form of glasses, provides view an augmented reality image including text, such as in an electronic book or magazine, word processing document, email, karaoke, teleprompter or other public speaking assistance application. The presentation of text and/or graphics can be adjusted based on sensor inputs indicating a gaze direction, focal distance and/or biological metric of the user. A current state of the text can be bookmarked when the user looks away from the image and subsequently resumed from the bookmarked state. A forward facing camera can adjust the text if a real word object passes in front of it, or adjust the appearance of the text based on a color of pattern of a real world background object. In a public speaking or karaoke application, information can be displayed regarding a level of interest of the audience and names of audience members.

Claims

1. A head-mounted display device, comprising:

a control circuit;

a microdisplay responsive to the control circuit, the microdisplay projects augmented reality images including text to an eye of a user; and

a sensor associated with the control circuit, the sensor tracks a gaze direction of the user, the control circuit, responsive to the sensor, controls the microdisplay to advance the text based on the gaze direction of the user, determines a reading speed of the user based on changes in the gaze direction of the user over time, the changes comprise saccades in which the gaze direction of the user moves along a line of the text and a return sweep saccade in which the gaze direction moves to a beginning of a next line of the text, and controls the microdisplay to advance the text based on the reading speed of the user.

3. The head-mounted display device of claim 1, wherein:

the control circuit;

determines a gaze pattern of the user based on changes in the gaze direction of the user over time, and determines a template from among a plurality of templates which is a closest match to the gaze pattern, one template of the plurality of templates is associated with a reading disability.

4. The head-mounted display device of claim 1, wherein:

the augmented reality images include the text on a virtual object; and

the sensor tracks an orientation of a head of the user, and the control circuit controls the microdisplay to adjust an orientation of the virtual object and the text, as the orientation of the head of the user changes, so that the virtual object and the text appear to the user to be registered to a real world environment.

5. The head-mounted display device of claim 1, wherein:

the augmented reality images include the text on a virtual object;

the sensor comprises a forward-facing camera having a field of view; and

the control circuit;

(i) uses images from the forward-facing camera to determine when a real world object passes between the eye of the user and the text on the virtual object, in the field of view, and (ii) controls the microdisplay to adjust the augmented reality images based on shape of the real world object, so that the augmented reality images do not appear to be rendered on at least a portion of the real world object.

6. The head-mounted display device of claim 1, wherein:

the sensor comprises a forward-facing depth camera having a field of view, the forward-facing depth camera determines at least one of a color or a pattern of a real world object in the field of view, and the control circuit controls the microdisplay to adjust an appearance of the text based on the at least one of the color or the pattern of the real world object.

7. The head-mounted display device of claim 1, wherein:

the augmented reality images include a system message; and

the control circuit determines a comprehension level of the user in reading the system message based on at least one of the gaze direction or a focal distance of the user, and determines a time to repeat the system message based on the comprehension level of the user in reading the system message.

8. The head-mounted display device of claim 1, wherein:

the control circuit determines the reading speed of the user based on a transition between gaze locations and a number of words encompassed by the transition.

9. The head-mounted display device of claim 1, wherein:

the control circuit determines the reading speed of the user in words per minute.

10. The head-mounted display device of claim 1, wherein:

the augmented reality images include a system message; and

the control circuit determines a comprehension level of the user in reading the system message based on at least one of the gaze direction or a focal distance of the user, and determines a new system message to display, based on the comprehension level of the user in reading the system message.

11. The head-mounted display device of claim 1, wherein:

the control circuit smooths the reading speed of the user.

2. The head-mounted display device of claimed 1, wherein:

the control circuit controls the microdisplay to display auxiliary information related to at least one of a word, a set of words or a graphic element in the augmented reality images, when the control circuit determines, base on the gaze direction, that the user gazes at the at least one of the words, the set of words or the graphic element for a threshold amount of time.

12. A head-mounted display device, comprising:

a control circuit;

a microdisplay responsive to the control circuit, the microdisplay projects augmented reality images including text to an eye of a user; and

a sensor associated with the control circuit, the sensor tracks a gaze direction and a focal distance of the user, the control circuit, responsive to the sensor, controls the microdisplay to advance the text based on the gaze direction of the user, and provides a bookmark of a state of the text when the focal distance of the user changes from a focal distance which is consistent with a focal distance of the text, to a focal distance which is inconsistent with the focal distance of the text.

13. The head-mounted display device of claim 12, wherein:

the control circuit controls the microdisplay to;

stop a display of the text when the focal distance of the user changes from the focal distance which is consistent with the focal distance of the text, to the focal distance which is inconsistent with the focal distance of the text.

14. The head-mounted display device of claim 13, wherein:

the control circuit controls the microdisplay to resume the display of the text based on the bookmark when the focal distance of the user returns to the focal distance which is consistent with the focal distance of the text.

15. A head-mounted display device, comprising:

a control circuit;

a microdisplay responsive to the control circuit, the microdisplay projects augmented reality images including a graphic element to an eye of a user; and

a sensor associated with the control circuit, the sensor tracks a gaze direction of the user, the control circuit, responsive to the sensor, controls the microdisplay to increase a size of the graphic element when the control circuit determines, based on the gaze direction, that the user gazes at the graphic element for a threshold amount of time.

16. The head-mounted display device of claim 15, wherein:

the graphic element comprises a 2d graphic element.

17. The head-mounted display device of claim 15, wherein:

the graphic element comprises a 3d graphic element.