Systems and methods for accessing a nested menu

• US 10,254,920 B2
• Filed: 12/01/2014
• Issued: 04/09/2019
• Est. Priority Date: 12/01/2013
• Status: Active Grant

• Current Assignee: Upskill, Inc. (TeamViewer AG)
• Original Assignee: Upskill, Inc. (TeamViewer AG)
• Inventors: Reily, Todd Richard, Tagunicar, Harold Ronald Villanueva, Sciscenti, Michael Anthony
• Primary Examiner(s): Tillery, Rashawn N
• Application Number:
  US14/556,556
• Publication Number:
  US 20150153912A1
• Time in Patent Office: 1,590 Days
• CPC Class Codes: A45C 11/00 Receptacles for purposes not provided for in groups A45C1/00-A45C9/00 specially adapted for toiletry or cosmetic equipment A45D29/20, A45D44/18; travelling sewing kits A45F3/48
• CPC Class Codes: A45C 2011/002 for portable handheld communication devices, e.g. mobile phone, pager, beeper, PDA, smart phone
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 : Scrolling or panning
• CPC Class Codes: G06F 3/0486 : Drag-and-drop
• CPC Class Codes: G06F 3/0487 : using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 : for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 12/1822 : Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 67/52 : specially adapted for the location of the user terminal
• CPC Class Codes: H04M 1/72454 : according to context-related or environment-related conditions
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 23/61 : based on recognised objects
• CPC Class Codes: H04N 23/631 : Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/635 : Region indicators; Field of view indicators
• CPC Class Codes: H04W 12/084 : using delegated authorisation, e.g. open authorisation [OAuth] protocol
• CPC Class Codes: H04W 4/08 : User group management
• CPC Class Codes: H04W 76/10 : Connection setup

Abstract

Systems and methods allow a user to interact with an augmented reality device. In one implementation, a wearable device for providing a virtual menu to a user includes a display; at least one sensor configured to provide an output indicative of a viewing direction of the user; and at least one processing device. The at least one processing device is configured to monitor a viewing direction of the user based on the output of the at least one sensor; provide a menu on the display; determine, based on the monitored viewing direction, whether the user is looking in a direction of the location of the menu on the display; and expand the menu if the user is determined to be looking in the direction of the location of the menu on the display.

Claims

1. A wearable device for providing a virtual menu to a user, the wearable device comprising:

a display;

a frame configured to secure the wearable device to a head of the user;

at least one sensor attached to the frame and configured to output an angle indicative of an orientation of the wearable device relative to a horizontal plane; and

at least one processing device configured to;

monitor the orientation of the wearable device based on the angle output by the at least one sensor;

cause the menu to appear on the display based on the angle output by the at least one sensor being greater than or equal to a predetermined threshold angle, wherein a location of the menu in a horizontal direction on the display is unaffected by horizontal movement of the wearable device;

cause an indicator of a predetermined dwell time threshold representing a time duration to be shown on the display;

determine, based on the monitored orientation, whether the user is looking in a direction of the location of the menu on the display;

in response to a determination that the user is looking in the direction of the location of the menu, monitor a time period that the user is looking in the direction of the location of the menu;

determine whether the orientation of the wearable device has changed from a first orientation to a second orientation;

in response to a determination that the orientation of the wearable device has changed from a first orientation to a second orientation, determine whether a difference between the first orientation and the second orientation is less than an orientation threshold;

in response to a determination that the difference between the first orientation and the second orientation is less than the orientation threshold, continue to monitor the time period that the user is looking in the direction of the location of the menu;

in response to a determination that the difference between the first orientation and the second orientation is not less than the orientation threshold, reset the time period that the user is looking in the direction of the location of the menu to zero; and

expand the menu if the user is determined to be looking in the direction of the location of the menu on the display and the monitored time period that the user is looking in the direction of the location of the menu exceeds the predetermined dwell time threshold.

2. The wearable device of claim 1, wherein expanding the menu includes showing to the user at least one menu element not shown upon initial display of the menu.

3. The wearable device of claim 1, wherein expanding the menu includes exploding the menu into a plurality of new menu items shown on the display.

4. The wearable device of claim 3, wherein the plurality of new menu items is arranged around an originally displayed menu item.

5. The wearable device of claim 1, wherein the at least one processing device is further configured to scroll through the menu in response to user input.

6. The wearable device of claim 5, wherein the user input includes an output of an eye tracking sensor configured to determine a direction of the user'"'"'s gaze.

7. The wearable device of claim 1, wherein the at least one processing device shows the menu on the display such that the user perceives the menu location as fixed relative to real world coordinates and variable relative to display coordinates.

8. The wearable device of claim 1, wherein the at least one processing device is configured to change a location of the menu relative to the display when the user rotates the head toward the menu.

9. The wearable device of claim 1, wherein the at least one processing device is configured to display a reticle on the display and expand the menu if the user moves a position of the reticle to intercept at least a portion of the menu.

10. The wearable device of claim 1, wherein the at least one sensor is configured to provide an output indicative of the viewing direction of the user by tracking a pattern of movement of an orientation of the head of the user.

11. The wearable device of claim 1, wherein the at least one sensor is configured to provide an output indicative of the viewing direction of the user by tracking a gaze of the user'"'"'s eyes.

25. The wearable device of claim 1, wherein the at least one processing device is configured to provide an interface for setting a dwell time threshold.

26. The wearable device of claim 1, wherein the at least one processing device is configured to provide a selectable option to switch between a locked mode and an unlocked mode, wherein the locked mode locks the menu to a particular location relative to the display and the unlocked mode allows the menu to move relative to the display.

12. A method for providing a virtual menu to a user of a wearable device, the method comprising:

monitoring, based on an angle output by at least one sensor, an orientation of the wearable device relative to a horizontal plane, the angle being indicative of the orientation of the wearable device relative to the horizontal plane, wherein the at least one sensor is attached to a frame which is configured to secure the wearable device to a head of the user;

causing the menu to appear on a display of the wearable device based on the angle output by the at least one sensor being greater than or equal to a predetermined threshold angle, wherein a location of the menu in a horizontal direction on the display is unaffected by horizontal movement of the wearable device;

causing an indicator of a predetermined dwell time threshold representing a time duration to be shown on the display;

determining, based on the monitored orientation, whether the user is looking in a direction of the location of the menu on the display;

in response to a determination that the user is looking in the direction of the location of the menu, monitoring a time period that the user is looking in the direction of the location of the menu;

determining whether the orientation of the wearable device has changed from a first orientation to a second orientation;

in response to a determination that the orientation of the wearable device has changed from a first orientation to a second orientation, determining whether a difference between the first orientation and the second orientation is less than an orientation threshold;

in response to a determination that the difference between the first orientation and the second orientation is less than the orientation threshold, continuing to monitor the time period that the user is looking in the direction of the location of the menu;

in response to a determination that the difference between the first orientation and the second orientation is not less than the orientation threshold, resetting the time period that the user is looking in the direction of the location of the menu to zero; and

expanding the menu if the user is determined to be looking in the direction of the location of the menu on the display and the monitored time period that the user is looking in the direction of the location of the menu exceeds the predetermined dwell time threshold.

13. The method of claim 12, wherein the wearable device includes the at least one sensor.

14. The method of claim 12, wherein expanding the menu includes showing to the user at least one menu element not shown upon initial display of the menu.

15. The method of claim 12, wherein expanding the menu includes exploding the menu into a plurality of new menu items shown on the display.

16. The method of claim 15, wherein the plurality of new menu items is arranged around an originally displayed menu item.

17. The method of claim 12, further comprising scrolling through the menu in response to user input.

18. The method of claim 17, wherein the user input includes an output of an eye tracking sensor configured to determine a direction of the user'"'"'s gaze.

19. The method of claim 12, wherein the menu is shown on the display such that the user perceives the menu location as fixed relative to real world coordinates and variable relative to display coordinates.

20. The method of claim 12, further comprising changing a location of the menu relative to the display when the user rotates the head toward the menu.

21. The method of claim 12, further comprising displaying a reticle on the display and expand the menu if the user moves a position of the reticle to intercept at least a portion of the menu.

22. The method of claim 12, wherein the monitored viewing direction includes a monitored pattern of movement of an orientation of the head of the user.

23. The method of claim 12, wherein the tracked viewing direction includes a tracked gaze of the user'"'"'s eyes.

27. The method of claim 12, further comprising providing an interface for setting a dwell time threshold.

28. The method of claim 12, further comprising providing a selectable option to switch between a locked mode and an unlocked mode, wherein the locked mode locks the menu to a particular location relative to the display and the unlocked mode allows the menu to move relative to the display.

24. A non-transitory computer-readable storage medium storing program instructions executable by at least one processor to perform a method for providing a virtual menu to a user of a wearable device, the method comprising:

monitoring, based on an angle output by at least one sensor, an orientation of the wearable device relative to a horizontal plane, the angle being indicative of the orientation of the wearable device relative to the horizontal plane, wherein the at least one sensor is attached to a frame which is configured to secure the wearable device to a head of the user;

causing the menu to appear on a display of the wearable device based on the angle output by the at least one sensor being greater than or equal to a predetermined threshold angle, wherein a location of the menu in a horizontal direction on the display is unaffected by horizontal movement of the wearable device;

causing an indicator of a predetermined dwell time threshold representing a time duration to be shown on the display;

determining, based on the monitored orientation, whether the user is looking in a direction of the location of the menu on the display;

in response to a determination that the user is looking in the direction of the location of the menu, monitoring a time period that the user is looking in the direction of the location of the menu;

determining whether the orientation of the wearable device has changed from a first orientation to a second orientation;

in response to a determination that the orientation of the wearable device has changed from a first orientation to a second orientation, determining whether a difference between the first orientation and the second orientation is less than an orientation threshold;

in response to a determination that the difference between the first orientation and the second orientation is less than the orientation threshold, continuing to monitor the time period that the user is looking in the direction of the location of the menu;

in response to a determination that the difference between the first orientation and the second orientation is not less than the orientation threshold, resetting the time period that the user is looking in the direction of the location of the menu to zero; and

expanding the menu if the user is determined to be looking in the direction of the location of the menu on the display and the monitored time period that the user is looking in the direction of the location of the menu exceeds the predetermined dwell time threshold.