User interface

• US 8,611,015 B2
• Filed: 11/22/2011
• Issued: 12/17/2013
• Est. Priority Date: 11/22/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Wheeler, Aaron Joseph, Gomez, Luis Ricardo Prada, Raffle, Hayes Solos
• Primary Examiner(s): HASAN, MOHAMMED A
• Application Number:
  US13/302,916
• Publication Number:
  US 20130128364A1
• Time in Patent Office: 756 Days
• Field of Search: 359630-633, 359/13, 351200-246
• US Class Current: 359/630
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G01S 17/66 Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A head-mounted display (HMD) may include an eye-tracking system, an HMD-tracking system and a display configured to display virtual images. The virtual images may present an augmented reality to a wearer of the HMD and the virtual images may adjust dynamically based on HMD-tracking data. However, position and orientation sensor errors may introduce drift into the displayed virtual images. By incorporating eye-tracking data, the drift of virtual images may be reduced. In one embodiment, the eye-tracking data could be used to determine a gaze axis and a target object in the displayed virtual images. The HMD may then move the target object towards a central axis. The HMD may also record data based on the gaze axis, central axis and target object to determine a user interface preference. The user interface preference could be used to adjust similar interactions with the HMD.

Claims

1. A method comprising:

displaying images on a display, the display having a central axis;

determining a gaze axis with respect to the central axis;

determining a target object in the displayed images based on the gaze axis; and

adjusting the displayed images on the display to move the target object towards the central axis, wherein adjusting the displayed images on the display comprises moving the target object at a tracking rate and adjusting the tracking rate based on a difference between the gaze axis and the central axis.

2. The method of claim 1, wherein determining the gaze axis comprises:

obtaining an eye pupil image;

determining the gaze axis from the eye pupil image.

3. The method of claim 1, wherein determining the gaze axis comprises:

obtaining a plurality of eye pupil images; and

determining the gaze axis from the plurality of eye pupil images.

4. The method of claim 1, wherein adjusting the displayed images on the display to move the target object towards the central axis comprises:

moving a set of displayed images towards the central axis, wherein the set of displayed images comprises the target object.

5. A head-mounted display (HMD), comprising:

a head-mounted support;

an optical system attached to the head-mounted support, wherein the optical system comprises a display having a central axis, wherein the display is configured to display images that are viewable from a viewing location;

an infrared light source configured to illuminate the viewing location with infrared light such that infrared light is reflected from the viewing location as reflected infrared light;

a camera configured to image the viewing location by collecting the reflected infrared light;

a sensor configured to generate sensor data that relates to motion of the HMD; and

a computer configured to determine a gaze axis based on one or more images of the viewing location obtained by the camera, control the display to display images based on the sensor data, determine a target object in the displayed images based on the gaze axis, and control the display to move the target object towards the central axis.

6. The HMD of claim 5, wherein the viewing location corresponds to a location of at least one eye of a wearer of the HMD.

7. The HMD of claim 5, wherein the computer is configured to determine a pupil location of the at least one eye of a wearer of the HMD from the received images.

8. The HMD of claim 5, wherein the optical system further comprises a visible light source.

9. The HMD of claim 8, wherein the display panel is configured to generate the virtual image by spatially modulating visible light from the visible light source.

10. The HMD of claim 5, wherein the infrared light source comprises an infrared light-emitting diode (LED).

11. The HMD of claim 5, wherein the camera comprises an infrared camera.

12. The HMD of claim 5, wherein the computer is further configured to record data based on the central axis, the gaze axis, the target object, and the displayed images, and to adjust the displayed images generated by the display panel based on the recorded data.

13. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions, the functions comprising:

controlling a display to display images, the display having a central axis;

determining a gaze axis with respect to the central axis;

determining a target object in the displayed images based on the gaze axis; and

controlling the display to adjust the displayed images so as to move the target object towards the central axis, wherein controlling the display to adjust the displayed images comprises moving the target object at a tracking rate and adjusting the tracking rate based on a difference between the gaze axis and the central axis.

14. The non-transitory computer readable medium of claim 13, wherein determining a gaze axis comprises:

obtaining an eye pupil image;

determining the gaze axis from the eye pupil image.

15. The non-transitory computer readable medium of claim 13, wherein determining the gaze axis comprises:

obtaining a plurality of eye pupil images; and

determining a gaze axis from the plurality of eye pupil images.

16. The non-transitory computer readable medium of claim 13, wherein the functions further comprise:

recording data based on the central axis, the gaze axis, the target object, and the displayed images, and adjusting the displayed images on the display based on the recorded data.

17. The method of claim 13, wherein controlling the display to adjust the displayed images so as to move the target object towards the central axis comprises:

controlling the display to move a set of displayed images towards the central axis, wherein the set of displayed images comprises the target object.

18. A method, comprising:

displaying images on a display of a head-mounted display (HMD), wherein the displayed images are viewable at a viewing location and the display comprises a central axis;

acquiring sensor data related to the motion of the HMD;

controlling the display to display images based on the sensor data;

determining a gaze axis based on one or more images of the viewing location obtained by a camera;

determining a target object in the displayed images based on the gaze axis; and

controlling the display to move the target object towards the central axis.

19. The method of claim 18, wherein determining the gaze axis comprises:

obtaining an eye pupil image;

determining the gaze axis from the eye pupil image.

20. The method of claim 18, wherein determining the gaze axis comprises:

obtaining a plurality of eye pupil images; and

determining the gaze axis from the plurality of eye pupil images.

21. The method of claim 18, wherein controlling the display to move the target object towards the central axis comprises:

moving the target object at a tracking rate.

22. The method of claim 21, wherein controlling the display to move the target object towards the central axis further comprises:

adjusting the tracking rate based on a difference between the gaze axis and the central axis.

23. The method of claim 18, further comprising recording data based on the central axis, the gaze axis, the target object, and the displayed images.

24. The method of claim 23, wherein controlling the display to move the target object towards the central axis is further based on the recorded data.

25. The method of claim 18, wherein controlling the display to move the target object towards the central axis comprises:

controlling the display to move a set of displayed images towards the central axis, wherein the set of displayed images comprises the target object.