MANIPULATING VIRTUAL OBJECTS WITH SIX DEGREE-OF-FREEDOM CONTROLLERS IN AN AUGMENTED AND/OR VIRTUAL REALITY ENVIRONMENT

• US 20180059777A1
• Filed: 08/17/2017
• Published: 03/01/2018
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: KOBAYASHI, Tony, DUBOIS, Paul
• Granted Patent: US 10,642,344 B2
• Time in Patent Office: Days
• CPC Class Codes: A63B 2022/0271 omnidirectional
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A system for combining six degree of freedom (6DOF) inputs, or positional and orientation inputs, from multiple 6DOF controllers in an augmented reality (AR) environment and/or a virtual reality (VR) environment is provided. In response to the detected movement of multiple controllers, and in particular, rotational movement of the multiple controllers providing user inputs in a virtual reality environment, the detected movements of the multiple controllers may be resolved to a single, common coordinate system to determine intended input to be applied to a selected virtual object in the virtual environment. The ability to resolve these inputs from multiple 6DOF controllers to a common coordinate system may provide a more natural user input mode, thus enhancing the user'"'"'s experience.

Claims

1. A method, comprising:

generating a three-dimensional (3D) virtual environment;

tracking a six-degree-of-freedom (6DOF) position and orientation of a first controller in a physical environment;

tracking a 6DOF position and orientation of a second controller in the physical environment;

detecting a selection of a virtual object displayed in the virtual environment;

detecting an input targeted for the selected virtual object based on a detected movement of the first controller in the physical environment with respect to a first coordinate system and a detected movement of the second controller in the physical environment with respect to a second coordinate system;

resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a command movement with respect to a common coordinate system; and

manipulating the selected virtual object in response to the command movement.

2. The method of claim 1, wherein resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a command movement with respect to a common coordinate system includes:

defining a first axis of the common coordinate system, the first axis extending from a central portion of the first controller to a central portion of the second controller;

defining a second axis of the common coordinate system, the second axis being selected from one of an X axis, a Y axis, or a Z axis of the first coordinate system; and

defining a third axis of the common coordinate system, the third axis being the cross product of the first axis and the second axis.

3. The method of claim 2, wherein defining the second axis of the common coordinate system includes:

determining which of the X axis, the Y axis or the Z axis of the first coordinate system is most orthogonal to the first axis; and

setting the most orthogonal of the X axis, the Y axis or the Z axis as the second axis of the common coordinate system.

4. The method of claim 3, wherein defining the first axis includes dynamically re-defining the first axis at sequential points in time as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment.

5. The method of claim 4, wherein defining the second axis includes dynamically re-defining the second axis at sequential points in time, as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment, and as the first axis is dynamically re-defined.

6. The method of claim 5, wherein defining the third axis includes dynamically re-defining the third axis at sequential points in time, as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment, and as the first axis and the second axis are dynamically re-defined.

7. The method of claim 2, wherein detecting an input targeted for the selected virtual object based on the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system.

8. The method of claim 7, wherein resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to the command movement with respect to the common coordinate system includes:

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.

9. The method of claim 2, wherein detecting an input targeted for the selected virtual object based on the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system includes at least one of:

detecting a movement of the first controller with respect to at least one of the X axis, the Y axis or the Z axis of the first coordinate system, and with respect to a position of the second controller;

or

detecting a movement of the second controller with respect to at least one of an X axis, a Y axis or a Z axis of the second coordinate system, and with respect to a position of the first controller.

10. The method of claim 9, wherein resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to the command movement with respect to the common coordinate system includes:

determining a selected direction of movement and a selected amount of movement with respect to the common coordinate system based on at least one of the detected movement of the first controller or the detected movement of the second controller; and

applying the selected amount of movement in the selected direction of movement to the manipulation of the selected virtual object.

11. The method of claim 1, wherein generating the 3D virtual environment includes displaying the virtual environment on a display of a head mounted display (HMD) device, the first and second controllers being handheld controllers that are operably coupled with the HIVID, such that physical movement of the first and second controllers in the physical environment corresponds to user commands to be executed with respect to a selected virtual object displayed in the virtual environment.

12. A method, comprising:

detecting a selection of a virtual object displayed in a virtual environment;

detecting an input targeted for the selected virtual object displayed in the virtual environment in response to a detected movement of a first controller with respect to a first coordinate system, and a detected movement of a second controller with respect to a second coordinate system, the first controller and the second controller operating in a physical environment;

resolving the detected movement of the first controller and the detected movement of the second controller to a command movement with respect to a common coordinate system; and

rotating the selected virtual object in the virtual environment in response to the command movement.

13. The method of claim 12, wherein resolving the detected movement of the first controller and the detected movement of the second controller to the command movement with respect to the common coordinate system includes:

defining a first axis of the common coordinate system, the first axis extending from a central portion of the first controller to a central portion of the second controller;

defining a second axis of the common coordinate system, including;

determining which of an X axis, a Y axis or a Z axis of the first coordinate system is most orthogonal to the first axis; and

setting the most orthogonal of the X axis, the Y axis or the Z axis of the first coordinate system as the second axis of the common coordinate system; and

defining a third axis of the common coordinate system, the third axis being the cross product of the first axis and the second axis.

14. The method of claim 13, wherein

defining the first axis includes dynamically re-defining the first axis at sequential points in time as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment,

defining the second axis includes dynamically re-defining the second axis at sequential points in time, as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment, and as the first axis is dynamically re-defined, and

defining the third axis includes dynamically re-defining the third axis at sequential points in time, as the first controller and the second controller move in the physical environment, and correspondingly with respect to the virtual environment, and as the first axis and the second axis are dynamically re-defined.

15. The method of claim 13, wherein detecting an input targeted for the selected virtual object in response to the detected movement of the first controller and the detected movement of the second controller includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system.

16. The method of claim 15, wherein resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to the command movement with respect to the common coordinate system includes:

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.

17. The method of claim 12, further comprising:

tracking a six-degree-of-freedom (6DOF) position and orientation of the first controller in the physical environment;

tracking a 6DOF position and orientation of the second controller in the physical environment;

detecting the movement of the first controller based on the 6DOF tracking of the first controller; and

detecting the movement of the second controller based on the 6DOF tracking of the second controller.

18. A computer program product embodied on a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, causes the processor to execute a method, the method comprising:

generating a three-dimensional (3D) virtual environment;

tracking a six-degree-of-freedom (6DOF) position and orientation of a first controller in a physical environment;

tracking a 6DOF position and orientation of a second controller in the physical environment;

detecting a selection of a virtual object displayed in the virtual environment;

detecting an input targeted for the selected virtual object based on a detected movement of the first controller in the physical environment with respect to a first coordinate system and a detected movement of the second controller in the physical environment with respect to a second coordinate system;

resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a command movement with respect to a common coordinate system; and

manipulating the selected virtual object in response to the command movement.

19. The computer program product of claim 18, wherein resolving the detected movement of the first controller and the detected movement of the second controller to the command movement with respect to the common coordinate system includes:

defining a first axis of the common coordinate system, the first axis extending from a central portion of the first controller to a central portion of the second controller;

defining a second axis of the common coordinate system, including;

determining which of an X axis, a Y axis or a Z axis of the first coordinate system is most orthogonal to the first axis; and

setting the most orthogonal of the X axis, the Y axis or the Z axis as the second axis of the common coordinate system; and

defining a third axis of the common coordinate system, the third axis being the cross product of the first axis and the second axis.

20. The computer program product of claim 19, wherein

detecting an input targeted for the selected virtual object in response to the detected movement of the first controller and the detected movement of the second controller includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system, and

wherein resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to the command movement with respect to the common coordinate system includes;

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.