Manipulating virtual objects with six degree-of-freedom controllers in an augmented and/or virtual reality environment

• US 10,642,344 B2
• Filed: 08/17/2017
• Issued: 05/05/2020
• Est. Priority Date: 08/23/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Kobayashi, Tony, Dubois, Paul
• Primary Examiner(s): Liu, Gordon G
• Application Number:
  US15/679,597
• Publication Number:
  US 20180059777A1
• Time in Patent Office: 992 Days
• Field of Search: 345419
• CPC Class Codes: A63B 2022/0271 omnidirectional
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09B 9/00 Simulators for teaching or training purposes for the use of weapons F41; computing aspects G06; protocols for games, networked simulations or virtual reality H04L67/131

Abstract

A system for combining six degree of freedom (6DOF) inputs, or positional and orientation inputs, from multiple 6DOF controllers in an augmented reality (AR) environment and/or a virtual reality (VR) environment is provided. In response to the detected movement of multiple controllers, and in particular, rotational movement of the multiple controllers providing user inputs in a virtual reality environment, the detected movements of the multiple controllers may be resolved to a single, common coordinate system to determine intended input to be applied to a selected virtual object in the virtual environment. The ability to resolve these inputs from multiple 6DOF controllers to a common coordinate system may provide a more natural user input mode, thus enhancing the user'"'"'s experience.

Claims

1. A method, comprising:

generating a three-dimensional (3D) virtual environment;

tracking a six-degree-of-freedom (6DOF) position and orientation of a first controller in a physical environment;

tracking a 6DOF position and orientation of a second controller in the physical environment;

detecting a selection of a virtual object displayed in the virtual environment;

detecting an input targeted for the selected virtual object, including detecting a movement of the first controller and a movement of the second controller in the physical environment at the same point in time, the detected movement of the first controller being with respect to a first coordinate system and the detected movement of the second controller being with respect to a second coordinate system;

dynamically resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a single 6DOF command movement with respect to a common coordinate system, including;

defining a first axis of the common coordinate system, the first axis extending from a central portion of the first controller to a central portion of the second controller;

defining a second axis of the common coordinate system, the second axis of the common coordinate system being selected from one of an X axis, a Y axis, or a Z axis of one of the first coordinate system or the second coordinate system that is most orthogonal to the first axis of the common coordinate system; and

defining a third axis of the common coordinate system, the third axis of the common coordinate system being the cross product of the first axis of the common coordinate system and the second axis of the common coordinate system; and

manipulating the selected virtual object in response to the single 6DOF command movement.

2. The method of claim 1, wherein

defining the first axis includes dynamically defining the first axis of the common coordinate system as the first controller and the second controller move;

and

defining the second axis includes dynamically defining the second axis of the common coordinate system in response to detected continued movement of the first controller and the second controller; and

defining the third axis includes dynamically defining the third axis of the common coordinate system, the third axis being the cross product of the dynamically defined first axis and the dynamically defined second axis.

3. The method of claim 2, wherein defining the first axis includes dynamically re-defining the first axis at sequential points in time in response to the continued movement of the first controller and the second controller.

4. The method of claim 3, wherein dynamically defining the second axis includes dynamically re-defining the second axis at sequential points in time, in response to the continued movement of the first controller and the second controller, and in response to the dynamic re-defining of the first axis.

5. The method of claim 4, wherein defining the third axis includes dynamically re-defining the third axis at sequential points in time in response to the continued movement of the first controller and the second controller, and in response to the dynamic re-defining of the first axis and the second axis.

6. The method of claim 2, wherein detecting the input targeted for the selected virtual object includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system.

7. The method of claim 6, wherein dynamically resolving the detected movement of the first controller and the second controller to the single 6DOF command movement with respect to the common coordinate system includes:

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected rotational movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.

8. The method of claim 2, wherein detecting the input targeted for the selected virtual object includes at least one of:

detecting a movement of the first controller with respect to at least one of the X axis, the Y axis or the Z axis of the first coordinate system, and with respect to a position of the second controller;

or

detecting a movement of the second controller with respect to at least one of an X axis, a Y axis or a Z axis of the second coordinate system, and with respect to a position of the first controller.

9. The method of claim 8, wherein dynamically resolving the detected movement of the first controller and the second controller to the single 6DOF command movement with respect to the common coordinate system includes:

determining a selected direction of movement and a selected amount of movement with respect to the common coordinate system based on at least one of the detected movement of the first controller or the detected movement of the second controller; and

applying the selected amount of movement in the selected direction of movement to the manipulation of the selected virtual object.

10. The method of claim 1, wherein generating the 3D virtual environment includes displaying the virtual environment on a display of a head mounted display (HMD) device, the first and second controllers being handheld controllers that are operably coupled with the HMD, such that physical movement of the first and second controllers in the physical environment corresponds to user commands to be executed with respect to a selected virtual object displayed in the virtual environment.

11. The method of claim 1, wherein detecting the input targeted for the selected virtual object includes detecting simultaneous movement of the first controller and the second controller in the physical environment.

12. A method, comprising:

detecting a selection of a virtual object displayed in a virtual environment;

detecting an input targeted for the selected virtual object displayed in the virtual environment in response to a detected movement of a first controller and a detected movement of second controller in a physical environment at the same point in time, the detected movement of the first controller being with respect to a first coordinate system, and the detected movement of a second controller being with respect to a second coordinate system;

dynamically resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a single 6DOF command movement with respect to a common coordinate system, including;

defining a first axis of the common coordinate system, the first axis of the common coordinate system extending from a central portion of the first controller to a central portion of the second controller; and

dynamically defining a second axis of the common coordinate system, the second axis of the common coordinate system being selected from one of an X axis, a Y axis, or a Z axis of one of the first coordinate system or the second coordinate system that is most orthogonal to the first axis of the common coordinate system; and

defining a third axis of the common coordinate system, the third axis of the common coordinate system being the cross product of the first axis of the common coordinate system and the second axis of the common coordinate system; and

rotating the selected virtual object in the virtual environment in response to the single 6DOF command movement.

13. The method of claim 12, wherein

defining the first axis includes dynamically re-defining the first axis at sequential points in time in response to detected continued movement of at least one of the first controller or the second controller,

dynamically defining the second axis includes dynamically re-defining the second axis at sequential points in time in response to the detected continued movement of the at least one of the first controller or the second controller, and in response to the dynamic re-defining of the first axis, and

defining the third axis includes dynamically re-defining the third axis at sequential points in time in response to the detected continued movement of the at least one of the first controller or the second controller, and in response to the dynamic re-defining of the first axis and the second axis.

14. The method of claim 12, wherein detecting the input targeted for the selected virtual object includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system.

15. The method of claim 14, wherein dynamically resolving the detected movement of the first controller and the second controller to the single 6DOF command movement with respect to the common coordinate system includes:

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.

16. The method of claim 12, further comprising:

tracking a six-degree-of-freedom (6DOF) position and orientation of the first controller in the physical environment;

tracking a 6DOF position and orientation of the second controller in the physical environment;

detecting the movement of the first controller based on the 6DOF tracking of the first controller; and

detecting the movement of the second controller based on the 6DOF tracking of the second controller.

17. The method of claim 12, wherein detecting the input targeted for the selected virtual object includes detecting simultaneous movement of the first controller and the second controller in the physical environment.

18. A computer program product embodied on a non-transitory computer readable medium, the computer readable medium having stored thereon a sequence of instructions which, when executed by a processor, causes the processor to execute a method, the method comprising:

generating a three-dimensional (3D) virtual environment;

tracking a six-degree-of-freedom (6DOF) position and orientation of a first controller in a physical environment;

tracking a 6DOF position and orientation of a second controller in the physical environment;

detecting a selection of a virtual object displayed in the virtual environment;

detecting an input targeted for the selected virtual object including detecting a movement of the first controller and the second controller in the physical environment at the same point in time, the detected movement of the first controller being with respect to a first coordinate system, and the detected movement of the second controller being with respect to a second coordinate system;

dynamically resolving the detected movement of the first controller with respect to the first coordinate system and the detected movement of the second controller with respect to the second coordinate system to a single 6DOF command movement with respect to a common coordinate system, including;

defining a first axis of the common coordinate system, the first axis of the common coordinate system extending from a central portion of the first controller to a central portion of the second controller; and

defining a second axis of the common coordinate system, the second axis of the common coordinate system being selected from one of an X axis, a Y axis, or a Z axis of one of the first coordinate system or the second coordinate system that is most orthogonal to the first axis of the common coordinate system; and

defining a third axis of the common coordinate system, the third axis of the common coordinate system being the cross product of the first axis and the second axis; and

manipulating the selected virtual object in response to the single 6DOF command movement.

19. The computer program product of claim 18, wherein detecting the input targeted for the selected virtual object in response to the detected movement of the first controller and the second controller includes at least one of:

detecting a rotational movement of the first controller about at least one of the X axis, the Y axis or the Z axis of the first coordinate system;

or

detecting a rotational movement of the second controller about at least one of an X axis, a Y axis or a Z axis of the second coordinate system, and

wherein dynamically resolving the detected movement of the first controller and the second controller to the command movement with respect to the common coordinate system includes;

determining a rotational command movement with respect to the common coordinate system based on at least one of the detected rotational movement of the first controller or the detected movement of the second controller; and

applying the rotational command movement to the manipulation of the selected virtual object.

20. The computer program product of claim 18, wherein detecting the input targeted for the selected virtual object includes detecting simultaneous movement of the first controller and the second controller in the physical environment.