Thumb scroll user interface element for augmented reality or virtual reality environments

• US 10,402,081 B1
• Filed: 08/28/2018
• Issued: 09/03/2019
• Est. Priority Date: 08/28/2018
• Status: Active Grant

• Current Assignee: FMR LLC
• Original Assignee: FMR LLC
• Inventors: Andersen, James, Schouela, Adam, Wang, Hangyu
• Primary Examiner(s): Fibbi, Christopher J
• Application Number:
  US16/115,461
• Time in Patent Office: 371 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Methods and apparatuses are described for providing a thumb scroll user interface element in a computerized visual environment. A viewing device displays a graphical user interface within a three-dimensional space that includes surface planes each associated with a graphical thumb scroll element. A sensor device captures a location of the user'"'"'s hand within the three-dimensional space and a gesture of the user'"'"'s hand. A computing device detects that the location is within a defined zone around one of the graphical thumb scroll elements and activates the thumb scroll element associated with the defined zone. The computing device identifies the gesture, determines a scroll speed based upon a speed of the identified gesture, and determines a scroll direction based upon a direction of the identified gesture. The computing device changes characteristics of graphical content in the surface plane based upon at least one of: the scroll speed or the scroll direction.

Claims

1. A system for providing a thumb scroll user interface element in a computerized visual environment, the system comprising:

a viewing device that displays, to a user of the viewing device, a graphical user interface within a three-dimensional space, the graphical user interface including one or more surface planes that comprise graphical content, each surface plane associated with a graphical thumb scroll element,

wherein the viewing device is a virtual reality (VR) headset, an augmented reality (AR) headset, or a mixed reality (MR) headset, and

wherein the graphical thumb scroll element is a cylindrical knob attached to the corresponding surface plane and capable of being turned;

a sensor device that captures a location of the user'"'"'s thumb and finger within the three-dimensional space and a gesture of the user'"'"'s thumb and finger; and

a computing device coupled to the viewing device and the sensor device that;

detects that the location of at least a portion of the user'"'"'s thumb and finger are within a defined zone around one of the graphical thumb scroll elements;

activates the graphical thumb scroll element associated with the defined zone;

identifies that the gesture of the user'"'"'s thumb and finger comprises a rotation of the thumb and finger around the cylindrical knob;

determines a scroll speed of the activated graphical thumb scroll element based upon a speed of the rotation of the thumb and finger around the cylindrical knob;

determines a scroll direction of the activated graphical thumb scroll element based upon a direction of the rotation of the thumb and finger around the cylindrical knob;

changing one or more characteristics of the graphical content in the surface plane of the graphical user interface that is associated with the activated graphical thumb scroll element, wherein the one or more characteristics of the graphical content change according to the determined scroll speed and the determined scroll direction.

2. The system of claim 1, wherein the sensor device is a camera.

3. The system of claim 1, wherein the computing device is a processor and a memory that are embedded in the viewing device or in the sensor device.

4. The system of claim 1, wherein changing one or more characteristics of the graphical content in the surface plane comprises moving the graphical content in a direction consistent with the scroll direction.

5. The system of claim 1, wherein changing one or more characteristics of the graphical content in the surface plane comprises moving the graphical content at a speed consistent with the scroll speed.

6. The system of claim 1, wherein changing one or more characteristics of the graphical content in the surface plane comprises increasing a value displayed in the surface plane or decreasing a value displayed in the surface plane.

7. The system of claim 1, wherein identifying the gesture of the user'"'"'s thumb and finger comprises determining an orientation of the user'"'"'s thumb and finger with respect to the activated graphical thumb scroll element.

8. The system of claim 7, wherein determining an orientation of the user'"'"'s thumb and finger comprises determining at least one of:

a position of the thumb and finger or a relationship between a position of the thumb and the finger on the user'"'"'s hand.

9. The system of claim 1, further comprising detecting that the location of at least one of the user'"'"'s thumb or finger is no longer within the defined zone, and deactivating the activated thumb scroll element.

10. The system of claim 1, further comprising generating one or more visual indicators in proximity to the activated graphical thumb scroll element in the graphical user interface.

11. The system of claim 10, wherein the one or more visual indicators are arrows indicating scrollable content in the surface plane associated with the activated graphical thumb scroll element.

12. A computerized method of providing a thumb scroll user interface element in a computerized visual environment, the method comprising:

displaying, by a viewing device to a user of the viewing device, a graphical user interface within a three-dimensional space, the graphical user interface including one or more surface planes that comprise graphical content, each surface plane associated with a graphical thumb scroll element,

wherein the viewing device is a virtual reality (VR) headset, an augmented reality (AR) headset, or a mixed reality (MR) headset, and

wherein the graphical thumb scroll element is a cylindrical knob attached to the corresponding surface plane and capable of being turned;

capturing, by a sensor device, a location of the user'"'"'s thumb and finger within the three-dimensional space and a gesture of the user'"'"'s thumb and finger;

detecting, by a computing device coupled to the viewing device and the sensor device, that the location of at least a portion of the user'"'"'s thumb and finger are within a defined zone around one of the graphical thumb scroll elements in the graphical user interface;

activating, by the computing device, the graphical thumb scroll element associated with the defined zone;

identifying, by the computing device, that the gesture of the user'"'"'s thumb and finger comprises a rotation of the thumb and finger around the cylindrical knob;

determining, by the computing device, a scroll speed of the activated graphical thumb scroll element based upon a speed of the rotation of the thumb and finger around the cylindrical knob;

determining, by the computing device, a scroll direction of the activated graphical thumb scroll element based upon a direction of the rotation of the thumb and finger around the cylindrical knob; and

changing one or more characteristics of the graphical content in the surface plane of the graphical user interface that is associated with the activated graphical thumb scroll element, wherein the one or more characteristics of the graphical content change according to;

the determined scroll speed and the determined scroll direction.

13. The method of claim 12, wherein the sensor device is a camera.

14. The method of claim 12, wherein the computing device is a processor and a memory that are embedded into the viewing device or the sensor device.

15. The method of claim 12, wherein changing one or more characteristics of the graphical content in the surface plane comprises moving the graphical content in a direction consistent with the scroll direction.

16. The method of claim 12, wherein changing one or more characteristics of the graphical content in the surface plane comprises moving the graphical content at a speed consistent with the scroll speed.

17. The method of claim 12, wherein changing one or more characteristics of the graphical content in the surface plane comprises increasing a value displayed in the surface plane or decreasing a value displayed in the surface plane.

18. The method of claim 12, wherein identifying the gesture of the user'"'"'s thumb and finger comprises determining an orientation of the user'"'"'s thumb and finger with respect to the activated graphical thumb scroll element.

19. The method of claim 18, wherein determining an orientation of the user'"'"'s thumb and finger comprises determining at least one of:

a position of the thumb and finger or a relationship between a position of the thumb and the finger on the user'"'"'s hand.

20. The method of claim 12, wherein the computing device detects that the location of at least one of the user'"'"'s thumb or finger is no longer within the defined zone, and deactivating the activated thumb scroll element.

21. The method of claim 12, wherein the computing device generates one or more visual indicators in proximity to the activated graphical thumb scroll element in the graphical user interface.

22. The method of claim 21, wherein the one or more visual indicators are arrows indicating scrollable content in the surface plane associated with the activated graphical thumb scroll element.