Automatic customization of graphical user interface for optical see-through head mounted display with user interaction tracking

• US 10,089,786 B2
• Filed: 01/09/2014
• Issued: 10/02/2018
• Est. Priority Date: 08/19/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Liu, Kexi, Rahman, Md Sazzadur, Renschler, Martin H., Forutanpour, Babak
• Primary Examiner(s): Harrison, Chante
• Application Number:
  US14/151,620
• Publication Number:
  US 20150049112A1
• Time in Patent Office: 1,727 Days
• Field of Search: 345173, 345178, 345156, 345619, 345179, 345 7, 345 8, 345418, 345629, 345633, 345660-661, 715862, 715863, 715700, 715767, 715821-823, 715860, 715812, 715858
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, an apparatus, and a computer program product render a graphical user interface (GUI) on an optical see-through head mounted display (HMD). The apparatus obtains a location on the HMD corresponding to a user interaction with a GUI object displayed on the HMD. The GUI object may be an icon on the HMD and the user interaction may be an attempt by the user to select the icon through an eye gaze or gesture. The apparatus determines whether a spatial relationship between the location of user interaction and the GUI object satisfies a criterion, and adjusts a parameter of the GUI object when the criterion is not satisfied. The parameter may be one or more of a size of the GUI object, a size of a boundary associated with the GUI object or a location of the GUI object.

Claims

1. A method comprising:

obtaining a first location corresponding to a user interaction with a display, wherein the first location is in a screen coordinate system of the display;

determining a first distance between the first location and a graphical user interface (GUI) object presented on the display;

comparing the first distance to a threshold distance;

initially adjusting a parameter of the GUI object as a result of the comparison, wherein initially adjusting the parameter comprises;

initially increasing the parameter of the GUI object when the first distance is greater than the threshold distance, and initially decreasing the parameter of the GUI object when the first distance is less than the threshold distance, wherein the parameter is unadjusted prior to adjustment; and

rendering the GUI object with the adjusted parameter for presentment on the display.

2. The method of claim 1, wherein the user interaction includes an eye gaze.

3. The method of claim 1, wherein the user interaction includes a gesture.

4. The method of claim 1, wherein determining the distance between the first location and the GUI object comprises:

determining the distance between the first location and a second location corresponding to the GUI object, wherein the second location is in the screen coordinate system of the display.

5. The method of claim 1, wherein determining the distance between the first location and the GUI object comprises:

determining the distance between the first location and a boundary associated with the GUI object.

6. The method of claim 5, wherein the boundary is defined by a geometric shape relative to the GUI object.

7. The method of claim 6, wherein the GUI object has a center, and wherein the geometric shape is defined relative to the center of the GUI object.

8. The method of claim 7, wherein the boundary has a center, wherein the center of the boundary corresponds to a second location in the screen coordinate system of the display, wherein a third location in the screen coordinate system of the display corresponds to the GUI object, and wherein the second location and the third location are the same.

9. The method of claim 8, wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

10. The method of claim 5, wherein the first distance is greater than the threshold distance when the first location lies outside the boundary.

11. The method of claim 5, wherein the first distance is less than the threshold distance when the first location lies inside the boundary.

12. The method of claim 5, wherein the boundary is a circle having a radius measured from a second location corresponding to the GUI object, wherein the second location is in the screen coordinate system of the display, and wherein the radius is a value such that the GUI object is within the boundary.

13. The method of claim 12, wherein the first distance is greater than the threshold distance when the first location lies outside the boundary.

14. The method of claim 12, wherein the first distance is less than the threshold distance when the first location lies inside the boundary.

15. The method of claim 12, wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

16. The method of claim 1, wherein the parameter comprises one or more of:

a size of the GUI object or a size of a boundary associated with the GUI object.

17. The method of claim 1, wherein the GUI object is a first GUI object of a plurality of GUI objects presented on the display, wherein a respective boundary corresponds to each respective GUI object of the plurality of GUI objects, and wherein the method further comprises:

determining that the first GUI object has been selected when the first location is closest to the boundary associated with the first GUI object.

18. The method of claim 1, wherein the GUI object is a first GUI object of a plurality of GUI objects presented on the display, wherein the plurality of GUI object includes a second GUI object, wherein a first boundary corresponds to the first GUI object and a second boundary corresponds to the second GUI object, and wherein the method further comprises:

determining, when the first location is equidistant from the first boundary and the second boundary, that the first GUI object has been selected based on one or more of;

the size of the first boundary and the size of the second boundary,

the size of the first GUI object and the size of the second GUI object, or

a number of times the first GUI object has been selected in the past and a number of times the second GUI object has been selected in the past.

19. The method of claim 1, wherein a device includes the display.

20. The method of claim 19, wherein the device with the display is a head mounted display (HMD).

21. The method of claim 1, wherein a second location in the screen coordinate system of the display corresponds to the GUI object, and wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

22. The method of claim 1, wherein initially increasing the parameter of the GUI object comprises:

initially increasing the parameter proportionally to the first distance.

23. The method of claim 1, wherein initially decreasing the parameter of the GUI object comprises:

initially decreasing the parameter proportionally to the first distance.

24. An apparatus comprising:

means for obtaining a first location corresponding to a user interaction with a display, wherein the first location is in a screen coordinate system of the display;

means for determining a distance between the first location and a graphical user interface (GUI) object presented on the display;

means for comparing the first distance to a threshold distance;

means for initially adjusting a parameter of the GUI object as a result of the comparison, wherein the means for initially adjusting the parameter is configured to;

initially increase the parameter of the GUI object when the first distance is greater than the threshold distance, and initially decrease the parameter of the GUI object when the first distance is less than the threshold distance, wherein the parameter is unadjusted prior to adjustment; and

means for rendering the GUI object with the adjusted parameter for presentment on the display.

25. The apparatus of claim 24, wherein the user interaction includes an eye gaze.

26. The apparatus of claim 24, wherein the user interaction includes a gesture.

27. The apparatus of claim 24, wherein the means for determining the distance between the first location and the GUI object is configured to:

determine the distance between the first location and a second location corresponding to the GUI object, wherein the second location is in the screen coordinate system of the display.

28. The apparatus of claim 24, wherein the means for determining the distance between the first location and the GUI object is configured to:

determine the distance between the first location and a boundary associated with the GUI object.

29. The apparatus of claim 28, wherein the boundary is defined by a geometric shape relative to the GUI object.

30. The apparatus of claim 29, wherein the GUI object has a center, and wherein the geometric shape is defined relative to the center of the GUI object.

31. The apparatus of claim 28, wherein the first distance is greater than the threshold distance when the first location lies outside the boundary.

32. The apparatus of claim 24, wherein the parameter comprises one or more of:

a size of the GUI object or a size of a boundary associated with the GUI object.

33. The apparatus of claim 24, wherein the GUI object is a first GUI object of a plurality of GUI objects presented on the display, wherein a respective boundary corresponds to each respective GUI object of the plurality of GUI objects, and wherein the apparatus further comprises:

means for determining that the first GUI object has been selected when the first location is closest to the boundary associated with the first GUI object.

34. The apparatus of claim 24, wherein the GUI object is a first GUI object of a plurality of GUI objects presented on the display, wherein the plurality of GUI object includes a second GUI object, wherein a first boundary corresponds to the first GUI object and a second boundary corresponds to the second GUI object, and wherein the apparatus further comprises:

means for determining, when the first location is equidistant from the first boundary and the second boundary, that the first GUI object has been selected based on one or more of;

the size of the first boundary and the size of the second boundary, the size of the first GUI object and the size of the second GUI object, or

a number of times the first GUI object has been selected in the past and a number of times the second GUI object has been selected in the past.

35. The apparatus of claim 24, wherein the apparatus is a head mounted apparatus.

36. The apparatus of claim 35, wherein the apparatus with the display is a head mounted display (HMD).

37. An apparatus comprising:

a memory;

a display; and

at least one processor coupled to the memory, wherein the at least one processor is configured to;

obtain a first location corresponding to a user interaction with the display, wherein the first location is in a screen coordinate system of the display;

determine a distance between the first location and a graphical user interface (GUI) object presented on the display;

compare the first distance to a threshold distance;

initially adjust a parameter of the GUI object as a result of the comparison, wherein to initially adjust the parameter, the at least one processor is configured to;

initially increase the parameter of the GUI object when the first distance is greater than the threshold distance, and initially decrease the parameter of the GUI object when the first distance is less than the threshold distance, wherein the parameter is unadjusted prior to adjustment; and

render the GUI object with the adjusted parameter for presentment on the display.

38. The apparatus of claim 37, wherein the user interaction includes an eye gaze.

39. The apparatus of claim 37, wherein the user interaction includes a gesture.

40. The apparatus of claim 37, wherein to determine the distance between the first location and the GUI object, the at least one processor is configured to:

determine the distance between the first location and a second location corresponding to the GUI object, wherein the second location is in the screen coordinate system of the display.

41. The apparatus of claim 40, wherein to determine the distance between the first location and the GUI object, the at least one processor is configured to:

determine the distance between the first location and a boundary associated with the GUI object.

42. The apparatus of claim 41, wherein the first distance is less than the threshold distance when the first location lies inside the boundary.

43. The apparatus of claim 41, wherein the boundary is a circle having a radius measured from a second location corresponding to the GUI object, wherein the second location is in the screen coordinate system of the display, and wherein the radius is a value such that the GUI object is within the boundary.

44. The apparatus of claim 43, wherein the first distance is greater than the threshold distance when the first location lies outside the boundary.

45. The apparatus of claim 43, wherein the first distance is less than the threshold distance when the first location lies inside the boundary.

46. The apparatus of claim 43, wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

47. The apparatus of claim 41, wherein the boundary is defined by a geometric shape relative to the GUI object, wherein the GUI object has a center, and wherein the geometric shape is defined relative to the center of the GUI object.

48. The apparatus of claim 47, wherein the boundary has a center, wherein the center of the boundary corresponds to a second location in the screen coordinate system of the display, wherein a third location in the screen coordinate system of the display corresponds to the GUI object, and wherein the second location and the third location are the same.

49. The apparatus of claim 48, wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

50. The apparatus of claim 37, wherein the apparatus is a head mounted apparatus.

51. The apparatus of claim 50, wherein the apparatus with the display is a head mounted display (HMD).

52. The apparatus of claim 37, wherein a second location in the screen coordinate system of the display corresponds to the GUI object, and wherein the GUI object has a center and the second location corresponds to the center of the GUI object.

53. The apparatus of claim 37, wherein to initially increase the parameter of the GUI object, the at least one processor is configured to:

initially increase the parameter proportionally to the first distance.

54. The apparatus of claim 37, wherein to initially decrease the parameter of the GUI object, the at least one processor is configured to:

initially decrease the parameter proportionally to the first distance.

55. A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause at least one processor of a device to:

obtain a first location corresponding to a user interaction with a display, wherein the first location is in a screen coordinate system of the display;

determine a distance between the first location and a graphical user interface (GUI) object presented;

compare the first distance to a threshold distance;

initially adjust a parameter of the GUI object as a result of the comparison, wherein the instructions, when executed, cause the at least one processor to;

initially increase the parameter of the GUI object when the first distance is greater than the threshold distance, and initially decrease the parameter of the GUI object when the first distance is less than the threshold distance, wherein the parameter is unadjusted prior to adjustment; and

render the GUI object with the adjusted parameter for presentment on the display.