USER INTERFACE INTERACTION FOR TRANSPARENT HEAD-MOUNTED DISPLAYS

• US 20130335303A1
• Filed: 03/13/2013
• Published: 12/19/2013
• Est. Priority Date: 06/14/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MACIOCCI, Giuliano, MULHOLLAND, James Joseph
• Granted Patent: US 9,389,420 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Embodiments of the present invention are directed toward enabling a user to quickly interact with a graphical user interface (GUI) displayed by the HMD. Utilizing techniques provided herein, a hand or other object can be used to select visual elements displayed by the HMD. The visual elements can be located within larger active regions, allowing the user to more easily select a visual element by selecting the active region in which the visual element is disposed.

Claims

1. A method of providing a user interface in a head-mounted display (HMD), the method comprising:

causing a first visual element to be displayed in a transparent display area of the HMD, the first visual element being selectable by a user of the HMD;

defining a first active region of the transparent display area, larger than the first visual element, such that the first visual element is disposed within a portion of the first active region;

tracking a physical object that appears, from a perspective of the user, within the transparent display area;

determining at least a portion of the physical object appears, from the user'"'"'s perspective, within the first active region, indicating a selection of the first visual element; and

manipulating, with a processing unit, the user interface, based on the selection.

2. The method of providing the user interface in the HMD as recited in claim 1, wherein a border of the first active region extends to an edge of the transparent display area.

3. The method of providing the user interface in the HMD as recited in claim 1, further comprising causing the first active region to be displayed in the transparent display area based on the tracking.

4. The method of providing the user interface in the HMD as recited in claim 1, wherein:

the first active region is one of a plurality of active regions;

the first visual element is one of a plurality of visual elements; and

the plurality of visual elements are displayed such that, for each active region of the plurality of active regions, a visual element of the plurality of visual elements is encompassed by a respective active region.

5. The method of providing the user interface in the HMD as recited in claim 4, further comprising causing each active region of the plurality of active regions to be displayed in the transparent display area.

6. The method of providing the user interface in the HMD as recited in claim 4, wherein the plurality of active regions comprises a series of columns disposed, from the user'"'"'s perspective, within the transparent display area.

7. The method of providing the user interface in the HMD as recited in claim 6, wherein the determining comprises determining that the physical object occupies, from the user'"'"'s perspective, at least the bottom half of a first column of the series of columns, the first active region comprising the first column.

8. The method of providing the user interface in the HMD as recited in claim 4, wherein the plurality of active regions comprises quadrants of the transparent display area.

9. The method of providing the user interface in the HMD as recited in claim 4, further comprising delaying the displaying the plurality of visual elements until after a determination that the physical object appears, from the user'"'"'s perspective, within the transparent display area for a threshold amount of time.

10. The method of providing the user interface in the HMD as recited in claim 1, wherein manipulating the user interface comprises:

causing the first visual element to be removed from the transparent display area and causing two or more other visual elements to be displayed in the transparent display area, and

replacing the first active region with two or more active regions, each encompassing one of the two or more other visual elements.

11. The method of providing the user interface in the HMD as recited in claim 1, wherein manipulating the user interface occurs after the at least the portion of the physical object is determined to appear, from the user'"'"'s perspective, within the first active region for at least a threshold amount of time.

12. The method of providing the user interface in the HMD as recited in claim 11, further comprising displaying an indicator, prior to manipulating the user interface, indicating a length of time the at least the portion of the physical object is determined to appear, from the user'"'"'s perspective, within the first active region.

13. The method of providing the user interface in the HMD as recited in claim 1, wherein the physical object comprises a hand of the user.

14. The method of providing the user interface in the HMD as recited in claim 1, further comprising determining the physical object is within a threshold distance from the HMD.

15. A head-mounted display (HMD) comprising:

a display having a transparent display area;

a sensor configured to provide data regarding objects viewable through the transparent display area by a user of the HMD; and

a processing unit communicatively coupled with the display and the sensor, the processing unit configured to perform functions including;

causing the display to show a first visual element in the transparent display area, the first visual element being selectable by the user;

defining a first active region of the transparent display area such that the first visual element is disposed within a portion of the first active region;

using the data from the sensor to track a physical object that appears, from a user'"'"'s perspective, within the transparent display area;

determining at least a portion of the physical object appears, from the user'"'"'s perspective, within the first active region, indicating a selection of the first visual element; and

manipulating one or more elements displayed on the display based on the selection.

16. The HMD as recited in claim 15, wherein the processing unit is configured to define a border of the first active region such that the border of the first active region extends to an edge of the transparent display area.

17. The HMD as recited in claim 15, wherein the processing unit is configured to cause the display to show the first active region based on the tracked physical object.

18. The HMD as recited in claim 15, wherein:

the first active region is one of a plurality of active regions;

the first visual element is one of a plurality of visual elements; and

the processing unit is configured to cause the display to show the plurality of visual elements such that, for each active region of the plurality of active regions, a visual element of the plurality of visual elements is encompassed by a respective active region.

19. The HMD as recited in claim 18, wherein the processing unit is configured to cause the display to show each active region of the plurality of active regions.

20. The HMD as recited in claim 18, wherein the processing unit is configured to cause the display to show the plurality of active regions such that the plurality of active regions comprise a series of columns disposed, from the user'"'"'s perspective, within the transparent display area.

21. The HMD as recited in claim 20, wherein the processing unit is configured to determine that at least the portion of the physical object appears within the first active region by determining that the physical object occupies, from the user'"'"'s perspective, at least the bottom half of a first column of the series of columns, the first active region comprising the first column.

22. The HMD as recited in claim 18, wherein the processing unit is configured to cause the display to show the plurality of active regions such that the plurality of active regions comprise quadrants of the transparent display area.

23. The HMD as recited in claim 18, wherein the processing unit is configured to manipulate the one or more elements by:

causing the first visual element to be removed from the transparent display area and causing two or more other visual elements to be displayed in the transparent display area; and

replacing the first active region with two or more active regions, each encompassing one of the two or more other visual elements.

24. The HMD as recited in claim 18, wherein the processing unit is configured to delay causing the display to show the plurality of visual elements until after the processing unit determines that the physical object appears, from the user'"'"'s perspective, within the transparent display area for a threshold amount of time.

25. The HMD as recited in claim 15, wherein the processing unit is configured to manipulate the one or more elements after the processing unit determines that at least the portion of the physical object appears, from the user'"'"'s perspective, within the first active region for at least a threshold amount of time.

26. The HMD as recited in claim 25, wherein the processing unit is further configured to cause the display to show an indicator, prior to manipulating the one or more elements, indicating a length of time the processing unit determines the at least the portion of the physical object appears, from the user'"'"'s perspective, within the first active region.

27. The HMD as recited in claim 15, wherein the processing unit is further configured to determine the physical object is within a threshold distance from the HMD.

28. A computer-readable storage medium encoded with instructions for causing a head-mounted display (HMD) to perform functions including:

causing a first visual element to be displayed in a transparent display area of the HMD, the first visual element being selectable by a user of the HMD;

defining a first active region of the transparent display area, larger than the first visual element, such that the first visual element is disposed within a portion of the first active region;

tracking a physical object that appears, from a user'"'"'s perspective, within the transparent display area;

determining at least a portion of the physical object appears, from the user'"'"'s perspective, within the first active region, indicating a selection of the first visual element; and

manipulating one or more elements displayed in the transparent display area based on the selection.

29. An apparatus comprising:

means for causing a first visual element to be displayed in a transparent display area of an HMD, the first visual element being selectable by a user of the HMD;

means for defining a first active region of the transparent display area such that the first visual element is disposed within a portion of the first active region;

means for tracking a physical object that appears, from a user'"'"'s perspective, within the transparent display area;

means for determining at least a portion of the physical object appears, from the user'"'"'s perspective, within the first active region, indicating a selection of the first visual element; and

means for manipulating, with a processing unit, one or more elements based on the selection.

30. The apparatus as recited in claim 29, wherein the means for defining the first active region include means for defining a border of the first active region such that the border of the first active region extends to an edge of the transparent display area.

31. The apparatus as recited in claim 29, further comprising means for causing the first active region to be displayed in the transparent display area based on the means for tracking.

32. The apparatus as recited in claim 29, wherein the first active region is one of a plurality of active regions, and the first visual element is one of a plurality of visual elements, the apparatus further comprising means for displaying the plurality of visual elements such that, for each active region of the plurality of active regions, a visual element of the plurality of visual elements is encompassed by a respective active region.

33. The apparatus as recited in claim 32, further including means for causing each active region of the plurality of active regions to be displayed in the transparent display area.

34. The apparatus as recited in claim 32, further including means for defining the plurality of active regions such that the plurality of active regions comprises a series of columns disposed, from the user'"'"'s perspective, within the transparent display area.

35. The apparatus as recited in claim 34, wherein the means for determining that the at least the portion of the physical object appears, from the user'"'"'s perspective, within the first active region include means for determining that the physical object occupies, from the user'"'"'s perspective, at least the bottom half of a first column of the series of columns, the first active region comprising the first column.

36. The apparatus as recited in claim 32, further including means for defining the plurality of active regions as quadrants of the transparent display area.

37. The apparatus as recited in claim 32, wherein the means for manipulating the one or more elements include means for:

causing the first visual element to be removed from the transparent display area and causing two or more other visual elements to be displayed in the transparent display area, and

replacing the first active region with two or more active regions, each encompassing one of the two or more other visual elements.

38. The apparatus as recited in claim 32, wherein the means for displaying the plurality of visual elements are configured to display the plurality of visual elements after determining that the physical object appears, from the user'"'"'s perspective, within the transparent display area for a threshold amount of time.

39. The apparatus as recited in claim 29, wherein the means for manipulating the one or more elements are configured to manipulate the one or more elements after the at least the portion of the physical object is determined to appear, from the user'"'"'s perspective, within the first active region for at least a threshold amount of time.

40. The apparatus as recited in claim 39, further comprising means for displaying an indicator, prior to manipulating the one or more elements, indicating a length of time the at least the portion of the physical object is determined to appear, from the user'"'"'s perspective, within the first active region.

41. The apparatus as recited in claim 29, further comprising means for determining the physical object is within a threshold distance from the HMD.