COMBINING GYROMOUSE INPUT AND TOUCH INPUT FOR NAVIGATION IN AN AUGMENTED AND/OR VIRTUAL REALITY ENVIRONMENT

• US 20170329419A1
• Filed: 12/15/2016
• Published: 11/16/2017
• Est. Priority Date: 05/11/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Dearman, David, Li, Chun Yat Frank
• Granted Patent: US 10,509,487 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204

Abstract

A system for combining a gyromouse input with a touch surface input in an augmented reality (AR) environment and/or a virtual reality (VR) environment, a virtual display of virtual items and/or features may be adjusted in response to movement of the gyromouse combined with touch inputs, or touch and drag inputs, received on a touch surface of the gyromouse. Use of the gyromouse in the AR/VR environment may allow touch screen capabilities to be accurately projected into a three dimensional virtual space, providing a controller having improved functionality and utility in the AR/VR environment, and enhancing the user'"'"'s experience.

Claims

1. A method, comprising:

generating, in a head mounted display device operating in an ambient environment, a virtual environment;

setting an anchor point in a virtual display of a virtual object displayed in the virtual environment in response to a selection input received from a controller that is operably coupled to the head mounted display device;

receiving, from the controller, a first input implementing a first input mode;

adjusting an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input, a virtual display area of the virtual display being less than a display area needed to display all of the plurality of virtual features simultaneously;

receiving, from the controller, a second input implementing a second input mode, the second input mode being different than the first input mode; and

determining whether to ignore the second input or to execute an action in the virtual environment response to the second input based on a set of previously defined rules.

2. The method of claim 1, setting an anchor point in a virtual display of a virtual object in response to a selection input received from a controller including:

detecting an intersection point of a virtual beam directed from the controller with a portion of the virtual display; and

setting the anchor point at the detected intersection point in response to the selection input, the selection input including a detected activation of a manipulation device of the controller.

3. The method of claim 1, wherein the first input mode is a touch input on a touch surface of the controller, and the second input mode is a physical movement of the controller, and wherein the selection input is detected in response to an activation of a manipulation device of the controller.

4. The method of claim 3, adjusting an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input including:

moving through the plurality of virtual features, including one of;

sequentially scrolling through the plurality of virtual features, in a direction corresponding to the first input;

or

jumping to one or more virtual features, of the plurality of virtual features, corresponding to the first input, the first input corresponding to a selection of a virtual content button, the virtual content button corresponding to a set point in a sequential listing of the plurality of virtual features; and

displaying a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the moving.

5. The method of claim 3, setting an anchor point in a virtual display of a virtual object in response to a selection input received from a controller including:

detecting an intersection point of a virtual beam directed from the controller with one of a plurality of vertical scroll arrows positioned in a vertical scroll bar of the virtual display, or with one of a plurality of horizontal scroll arrows positioned in a horizontal scroll bar of the virtual display, or within a designated area of the vertical scroll bar, or within a designated area of the horizontal scroll bar;

setting the anchor point corresponding to the detected intersection point; and

scrolling through the plurality of virtual features associated with the virtual display in response to a touch input on the touch surface of the controller, the scrolling progressing in a direction corresponding to the anchor point and the touch input.

6. The method of claim 5, scrolling through the plurality of virtual features including:

sequentially scrolling through the plurality of virtual features in response to the touch input;

maintaining the scrolling while the touch input on the touch surface of the controller is maintained; and

terminating the scrolling when a release of the touch on the touch surface of the controller is detected.

7. The method of claim 5, the touch input on the touch surface of the controller being a series of tap inputs on the touch surface of the controller, the scrolling through the plurality of virtual features including sequentially scrolling, by a defined amount, through the plurality of virtual features in response to each tap input detected on the touch surface.

8. The method of claim 3, setting an anchor point in a virtual display in response to a selection input received from a controller including:

detecting an intersection point of a virtual beam directed from the controller with one of a top portion of the virtual display, a bottom portion of the virtual display, a left portion of the virtual display, a right portion of the virtual display, a vertical movement bar positioned in a vertical scroll bar of the virtual display, or a horizontal movement bar positioned in a horizontal scroll bar of the virtual display, and setting the anchor point at the detected intersection point in response to a touch input on the touch surface of the controller;

scrolling through the plurality of virtual features associated with the virtual object in response to a drag input on the touch surface, the plurality of virtual features displayed on the virtual display scrolling from the anchor point in a direction corresponding to a direction of the drag input on the touch surface of the controller; and

terminating the scrolling in response to a detected release of the touch and drag input on the touch surface of the controller.

9. The method of claim 3, wherein determining whether to ignore the second input or execute an action in the virtual environment response to the second input based on a set of previously defined rules includes:

ignoring the second input when the physical movement of the controller is within a previously defined range; and

executing an action in the virtual environment in response to the second input when the physical movement of the controller extends beyond the previously defined range.

10. The method of claim 3, wherein adjusting an arrangement of the plurality of virtual features associated with the virtual display of the virtual object in response to the first input includes scrolling through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the scrolling.

11. The method of claim 10, scrolling through the plurality of virtual features in response to the first input also including:

detecting a release of the activation of the manipulation device of the controller; and

terminating the scrolling of the plurality of virtual features in response to the detected release.

12. The method of claim 1, wherein the first input mode is a physical movement of the controller, and the second input mode is a touch input on a touch surface of the controller, and wherein the selection input is detected in response to an activation of a manipulation device of the controller.

13. The method of claim 12, wherein determining whether to ignore the second input or to execute an action in the virtual environment response to the second input based on a set of previously defined rules includes:

ignoring the second input when the touch input is contained within a previously defined area of the touch surface of the controller or a magnitude of the second input is less than a previously defined threshold; and

executing the action in the virtual environment response to the second input when the touch input extends beyond the previously defined area of the touch surface of the controller or the magnitude of the second input is greater than or equal to the previously defined threshold.

14. The method of claim 12, adjusting an arrangement of a plurality virtual objects associated with the virtual display of the virtual object in response to the first input including:

scrolling through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the scrolling;

detecting a release of the activation of the manipulation device of the controller; and

terminating the scrolling of the plurality of virtual objects in response to the detected release.

15. A computing device, comprising:

a memory storing executable instructions; and

a processor configured to execute the instructions, to cause the computing device to;

display a virtual environment on a display of a head mounted electronic device worn by a user and operating in an ambient environment;

set an anchor point in a virtual display of a virtual object displayed in the virtual environment in response to detecting selection input received from a controller that is operably coupled to the head mounted electronic device;

receive, from the controller, a first input implementing a first input mode;

adjust an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input, a virtual display area of the virtual display being less than a display area needed to display all of the plurality of virtual features simultaneously;

receive, from the controller, a second input implementing a second input mode, the second input mode being different than the first input mode; and

determining whether to ignore the second input or to execute an action in the virtual environment response to the second input based on a set of previously defined rules.

16. The computing device of claim 15, wherein the first input mode is a touch input on a touch surface of the controller, and the second input mode is a physical movement of the controller, and wherein, in setting the anchor point, the instructions cause the computing device to:

detect an intersection point of a virtual beam directed from the controller with a portion of the virtual display; and

set the anchor point at the detected intersection point in response to the selection input, the selection input being detected in response to a detected activation of a manipulation device of the controller.

17. The computing device of claim 16, wherein, in determining whether to ignore the second input or execute an action in the virtual environment response to the second input based on a set of previously defined rules, the instructions cause the computing device to:

ignore the second input when the physical movement of the controller is within a previously defined range; and

execute the action in the virtual environment response to the second input when the physical movement of the controller extends beyond the previously defined range.

18. The computing device of claim 16, wherein, in adjusting the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, the instructions cause the computing device to:

move through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the movement of the plurality of virtual features.

19. The computing device of claim 18, wherein, in moving through the plurality of virtual features, the instructions cause the computing device to:

sequentially scroll through the plurality of virtual features associated with the virtual display of the virtual object, in the direction corresponding to the touch input on the touch surface of the controller, the touch input;

terminate the scrolling in response to a detected release of the touch input on the touch surface of the controller; and

display a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the scrolling.

20. The computing device of claim 18, wherein, in moving through the plurality of virtual features, the instructions cause the computing device to:

jump to one or more virtual features, of the plurality of virtual features, corresponding to the first input, the first input corresponding to a selection of a virtual content button, the virtual content button corresponding to a set point in a sequential listing of the plurality of virtual features; and

display a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the jump.