Combining gyromouse input and touch input for navigation in an augmented and/or virtual reality environment

• US 10,509,487 B2
• Filed: 12/15/2016
• Issued: 12/17/2019
• Est. Priority Date: 05/11/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Dearman, David, Li, Chun Yat Frank
• Primary Examiner(s): Chowdhury, Rayeez R
• Application Number:
  US15/379,770
• Publication Number:
  US 20170329419A1
• Time in Patent Office: 1,097 Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204

Abstract

A system for combining a gyromouse input with a touch surface input in an augmented reality (AR) environment and/or a virtual reality (VR) environment, a virtual display of virtual items and/or features may be adjusted in response to movement of the gyromouse combined with touch inputs, or touch and drag inputs, received on a touch surface of the gyromouse. Use of the gyromouse in the AR/VR environment may allow touch screen capabilities to be accurately projected into a three dimensional virtual space, providing a controller having improved functionality and utility in the AR/VR environment, and enhancing the user'"'"'s experience.

Claims

1. A method, comprising:

generating, in a head mounted display device operating in an ambient environment, a virtual environment;

setting an anchor point in a virtual display of a virtual object displayed in the virtual environment in response to a selection input received from a handheld controller that is operably coupled to the head mounted display device;

receiving, from the controller, a first input implementing a first input mode of the handheld controller;

adjusting an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input, a virtual display area of the virtual display being less than a display area needed to display all of the plurality of virtual features simultaneously;

receiving, from the handheld controller, a second input implementing a second input mode of the handheld controller, the second input mode being different than the first input mode; and

executing a response to the second input from the handheld controller, based on a set of previously defined rules, including;

ignoring the second input in response to a determination that the second input includes a detected physical movement of the handheld controller that is inside a previously defined range; and

executing an action in the virtual environment in response to a determination that the second input includes a detected physical movement of the handheld controller that is outside of the previously defined range.

2. The method of claim 1, setting the anchor point in the virtual display of the virtual object in response to the selection input received from the controller including:

detecting an intersection point of a virtual beam directed from the controller with a portion of the virtual display; and

setting the anchor point at the detected intersection point in response to the selection input, the selection input including a detected activation of a manipulation device of the controller.

3. The method of claim 1, wherein the first input mode is a touch input on the touch surface of the controller, and the second input mode is a physical movement of the controller, and wherein the selection input is detected in response to an activation of a manipulation device of the controller.

4. The method of claim 3, adjusting the arrangement of the plurality of virtual features associated with the virtual display of the virtual object in response to the first input including:

moving through the plurality of virtual features, including;

sequentially scrolling through the plurality of virtual features, in a direction corresponding to the first input in response to a determination that the first input corresponds to a selection of a scrolling button of the virtual display; and

jumping to one or more virtual features, of the plurality of virtual features in response to a determination that the first input corresponds to a selection of a virtual content button, the virtual content button corresponding to a set point in a sequential listing of the plurality of virtual features; and

displaying a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the moving.

5. The method of claim 3, setting the anchor point in the virtual display of the virtual object in response to the selection input received from the controller including:

detecting an intersection point of a virtual beam directed from the controller with one of a plurality of vertical scroll arrows positioned in a vertical scroll bar of the virtual display, or with one of a plurality of horizontal scroll arrows positioned in a horizontal scroll bar of the virtual display, or within a designated area of the vertical scroll bar, or within a designated area of the horizontal scroll bar;

setting the anchor point corresponding to the detected intersection point; and

scrolling through the plurality of virtual features associated with the virtual display in response to a touch input on the touch surface of the controller, the scrolling progressing in a direction corresponding to the anchor point and the touch input.

6. The method of claim 5, scrolling through the plurality of virtual features including:

sequentially scrolling through the plurality of virtual features in response to the touch input;

maintaining the scrolling while the touch input on the touch surface of the controller is maintained; and

terminating the scrolling when a release of the touch on the touch surface of the controller is detected.

7. The method of claim 5, the touch input on the touch surface of the controller being a series of tap inputs on the touch surface of the controller, the scrolling through the plurality of virtual features including sequentially scrolling, by a defined amount, through the plurality of virtual features in response to each tap input detected on the touch surface.

8. The method of claim 3, setting the anchor point in the virtual display in response to the selection input received from the controller including:

detecting an intersection point of a virtual beam directed from the controller with one of a plurality of portions of the virtual display, plurality of portions of the virtual display including a top portion of the virtual display, a bottom portion of the virtual display, a left portion of the virtual display, a right portion of the virtual display, a vertical movement bar positioned in a vertical scroll bar of the virtual display, and a horizontal movement bar positioned in a horizontal scroll bar of the virtual display, and setting the anchor point at the detected intersection point in response to a touch input on the touch surface of the controller;

scrolling through the plurality of virtual features associated with the virtual object in response to a drag input on the touch surface, the plurality of virtual features displayed on the virtual display scrolling from the anchor point in a direction corresponding to a direction of the drag input on the touch surface of the controller; and

terminating the scrolling in response to a detected release of the touch and drag input on the touch surface of the controller.

9. The method of claim 3, wherein adjusting the arrangement of the plurality of virtual features associated with the virtual display of the virtual object in response to the first input includes scrolling through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the scrolling.

10. The method of claim 9, scrolling through the plurality of virtual features in response to the first input also including:

detecting a release of the activation of the manipulation device of the controller; and

terminating the scrolling of the plurality of virtual features in response to the detected release.

11. The method of claim 1, wherein the first input mode is a physical movement of the controller, and the second input mode is a touch input on a touch surface of the controller, and wherein the selection input is detected in response to an activation of a manipulation device of the controller.

12. The method of claim 11, adjusting the arrangement of the plurality virtual objects associated with the virtual display of the virtual object in response to the first input including:

scrolling through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the scrolling;

detecting a release of the activation of the manipulation device of the controller; and

terminating the scrolling of the plurality of virtual objects in response to the detected release.

19. The method of claim 1, wherein executing the response to the second input from the handheld controller, based on the set of previously defined rules, also includes:

executing an action in the virtual environment corresponding to the second input in response to a determination that the second input includes a detected touch input on a touch surface of the handheld controller that is outside of a previously defined area of the touch surface of the handheld controller; and

ignoring the second input in response to a determination that the second input includes a detected input on the touch surface of the handheld controller that is inside of the previously defined area of the touch surface of the handheld controller.

13. A computing device, comprising:

a memory storing executable instructions; and

a processor configured to execute the instructions, to cause the computing device to;

display a virtual environment on a display of a head mounted electronic device worn by a user and operating in an ambient environment;

set an anchor point in a virtual display of a virtual object displayed in the virtual environment in response to detecting selection input received from a handheld controller that is operably coupled to the head mounted electronic device;

receive, from the handheld controller, a first input implementing a first input mode;

adjust an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input, a virtual display area of the virtual display being less than a display area needed to display all of the plurality of virtual features simultaneously;

receive, from the handheld controller, a second input implementing a second input mode, the second input mode being different than the first input mode; and

execute a response to the second input based on a set of previously defined rules, including;

ignore the second input in response to a determination that the second input includes a detected touch input on a touch surface of the handheld controller that is inside a previously defined area of the touch surface of the handheld controller; and

execute an action in the virtual environment in response to a determination that the second input includes a touch on the touch surface of the handheld controller that is outside of the previously defined area of the touch surface of the handheld controller.

14. The computing device of claim 13, wherein the first input mode is a touch input on a touch surface of the controller, and the second input mode is a physical movement of the controller, and wherein, in setting the anchor point, the instructions cause the computing device to:

detect an intersection point of a virtual beam directed from the controller with a portion of the virtual display; and

set the anchor point at the detected intersection point in response to the selection input, the selection input being detected in response to a detected activation of a manipulation device of the controller.

15. The computing device of claim 14, wherein, in adjusting the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, the instructions cause the computing device to:

move through the plurality of virtual features associated with the virtual display of the virtual object in response to the first input, in a direction corresponding to the touch input on the touch surface of the controller, such that different combinations of virtual features of the plurality of virtual features are displayed in response to the movement of the plurality of virtual features.

16. The computing device of claim 15, wherein, in moving through the plurality of virtual features, the instructions cause the computing device to:

sequentially scroll through the plurality of virtual features associated with the virtual display of the virtual object, in the direction corresponding to the touch input on the touch surface of the controller, the touch input;

terminate the scrolling in response to a detected release of the touch input on the touch surface of the controller; and

display a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the scrolling.

17. The computing device of claim 15, wherein, in moving through the plurality of virtual features, the instructions cause the computing device to:

jump to one or more virtual features, of the plurality of virtual features, corresponding to the first input, the first input corresponding to a selection of a virtual content button, the virtual content button corresponding to a set point in a sequential listing of the plurality of virtual features; and

display a combination of virtual features, of the plurality of virtual features, on the virtual display of the virtual object, the displayed combination of virtual features being different from a combination of virtual features, of the plurality of virtual features, displayed before the jump.

18. A method, comprising:

generating, in a head mounted display device operating in an ambient environment, a virtual environment;

setting an anchor point in a virtual display of a virtual object displayed in the virtual environment in response to a selection input received from a handheld controller that is operably coupled to the head mounted display device;

receiving, from the controller, a first input implementing a first input mode of the handheld controller;

adjusting an arrangement of a plurality of virtual features associated with the virtual display of the virtual object in response to the first input, a virtual display area of the virtual display being less than a display area needed to display all of the plurality of virtual features simultaneously;

receiving, from the handheld controller, a second input implementing a second input mode of the handheld controller, the second input mode being different than the first input mode; and

executing a response to the second input from the handheld controller, based on a set of previously defined rules, including;

executing an action in the virtual environment corresponding to the second input in response to a determination that the second input includes a detected touch input on a touch surface of the handheld controller that is outside of a previously defined area of the touch surface of the handheld controller; and

ignoring the second input in response to a determination that the second input includes a detected input on the touch surface of the handheld controller that is inside the previously defined area of the touch surface of the handheld controller.