AUTOMATIC CUSTOMIZATION OF GRAPHICAL USER INTERFACE FOR OPTICAL SEE-THROUGH HEAD MOUNTED DISPLAY WITH USER INTERACTION TRACKING

• US 20150049112A1
• Filed: 01/09/2014
• Published: 02/19/2015
• Est. Priority Date: 08/19/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: LIU, Kexi, FORUTANPOUR, Babak, RAHMAN, Md Sazzadur, RENSCHLER, Martin H.
• Granted Patent: US 10,089,786 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, an apparatus, and a computer program product render a graphical user interface (GUI) on an optical see-through head mounted display (HMD). The apparatus obtains a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD. The GUI object may be an icon on the HMD and the user interaction may be an attempt by the user to select the icon through an eye gaze or gesture. The apparatus determines whether a spatial relationship between the location of user interaction and the GUI object satisfies a criterion, and adjusts a parameter of the GUI object when the criterion is not satisfied. The parameter may be one or more of a size of the GUI object, a size of a boundary associated with the GUI object or a location of the GUI object.

Claims

1. A method of rendering a graphical user interface (GUI) on an optical see-through head mounted display (HMD), said method comprising:

obtaining a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD;

determining whether a spatial relationship between the location and the GUI object satisfies a criterion; and

adjusting a parameter of the GUI object when the criterion is not satisfied.

2. The method of claim 1, wherein the user interaction is an eye gaze and obtaining a location on the HMD comprises obtaining data corresponding to the location from an eye tracking system that tracks user eye gaze.

3. The method of claim 1, wherein the user interaction is gesture and obtaining a location on the HMD comprises obtaining data corresponding to the location from a gesture tracking system that tracks user gestures.

4. The method of claim 1, wherein:

the location is defined by a location point on the HMD;

determining comprises obtaining data corresponding to an object point of the GUI object on the HMD; and

the spatial relationship is a distance between the location point and the object point.

5. The method of claim 4, wherein determining further comprises determining data corresponding to a boundary of the object.

6. The method of claim 5, wherein the object point is the center of the object.

7. The method of claim 6, wherein the boundary is a geometric shape defined relative to the center.

8. The method of claim 5, wherein the criterion is not satisfied when the distance is such that the location point lies outside the boundary.

9. The method of claim 1, wherein the parameter comprises one or more of a size of the GUI object, a size of a boundary associated with the GUI object, and a location of the GUI object.

10. The method of claim 1, wherein a parameter of each of a plurality of GUI objects has been adjusted and further comprising determining one of the GUI objects has been selected by a user interaction based on a location point corresponding to the user interaction.

11. The method of claim 10, wherein each GUI object has an associated boundary and the GUI object with the boundary closest to the location point is determined to have been selected.

12. The method of claim 11, wherein when the location point is equidistant from two or more boundaries, the determining is further based on one or more of the size of the respective boundaries, the size of the respective GUI objects, and the number of times the respective GUI objects have been selected in the past.

13. An apparatus for rendering a graphical user interface (GUI) on an optical see-through head mounted display (HMD), said apparatus comprising:

means for obtaining a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD;

means for determining whether a spatial relationship between the location and the GUI object satisfies a criterion; and

means for adjusting a parameter of the GUI object when the criterion is not satisfied.

14. The apparatus of claim 13, wherein the user interaction is an eye gaze and the means for obtaining a location on the HMD is configured to obtain data corresponding to the location from an eye tracking system that tracks user eye gaze.

15. The apparatus of claim 13, wherein the user interaction is gesture and the means for obtaining a location on the HMD is configured to obtain data corresponding to the location from a gesture tracking system that tracks user gestures.

16. The apparatus of claim 13, wherein:

the location is defined by a location point on the HMD;

the means for determining is configured to obtain data corresponding to an object point of the GUI object on the HMD; and

the spatial relationship is a distance between the location point and the object point.

17. The apparatus of claim 16, wherein the means for determining is further configured to determine data corresponding to a boundary of the object.

18. The apparatus of claim 17, wherein the object point is the center of the object.

19. The apparatus of claim 18, wherein the boundary is a geometric shape defined relative to the center.

20. The apparatus of claim 17, wherein the criterion is not satisfied when the distance is such that the location point lies outside the boundary.

21. The apparatus of claim 13, wherein the parameter comprises one or more of a size of the GUI object, a size of a boundary associated with the GUI object, and a location of the GUI object.

22. The apparatus of claim 13, wherein a parameter of each of a plurality of GUI objects has been adjusted and further comprising means for determining one of the GUI objects has been selected by a user interaction based on a location point corresponding to the user interaction.

23. The apparatus of claim 22, wherein each GUI object has an associated boundary and the GUI object with the boundary closest to the location point is determined to have been selected.

24. The apparatus of claim 23, wherein when the location point is equidistant from two or more boundaries, the means for determining determines further based on one or more of the size of the respective boundaries, the size of the respective GUI objects, and the number of times the respective GUI objects have been selected in the past.

25. An apparatus for rendering a graphical user interface (GUI) on an optical see-through head mounted display (HMD), said apparatus comprising:

a memory; and

at least processor coupled to the memory and configured to;

obtain a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD;

determine whether a spatial relationship between the location and the GUI object satisfies a criterion; and

adjusting a parameter of the GUI object when the criterion is not satisfied.

26. The apparatus of claim 25, wherein the user interaction is an eye gaze and the processor obtains a location on the HMD by being further configured to obtain data corresponding to the location from an eye tracking system that tracks user eye gaze.

27. The apparatus of claim 25, wherein the user interaction is gesture and the processor obtains a location on the HMD by being further configured to obtain data corresponding to the location from a gesture tracking system that tracks user gestures.

28. The apparatus of claim 25, wherein:

the location is defined by a location point on the HMD;

the processor determines by being configured to obtain data corresponding to an object point of the GUI object on the HMD; and

the spatial relationship is a distance between the location point and the object point.

29. The apparatus of claim 28, wherein the processor determines by being further configured to determine data corresponding to a boundary of the object.

30. A computer program product for rendering a graphical user interface (GUI) on an optical see-through head mounted display (HMD), said product comprising:

a computer-readable medium comprising code for;

obtaining a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD;

determining whether a spatial relationship between the location and the GUI object satisfies a criterion; and

adjusting a parameter of the GUI object when the criterion is not satisfied.