CONTROLLER TRACKING FOR MULTIPLE DEGREES OF FREEDOM

• US 20190033988A1
• Filed: 07/26/2017
• Published: 01/31/2019
• Est. Priority Date: 07/26/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hesch, Joel, Chen, Shiqi, Lee, Johnny, Garg, Rahul
• Granted Patent: US 10,852,847 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A method for controller tracking with multiple degrees of freedom includes generating depth data at an electronic device based on a local environment proximate the electronic device. A set of positional data is generated for at least one spatial feature associated with a controller based on a pose of the electronic device, as determined using the depth data, relative to the at least one spatial feature associated with the controller. A set of rotational data is received that represents three degrees-of-freedom (3DoF) orientation of the controller within the local environment, and a six degrees-of-freedom (6DoF) position of the controller within the local environment is tracked based on the set of positional data and the set of rotational data.

Claims

1. A method, comprising:

generating depth data at an electronic device based on a local environment proximate the electronic device;

generating a set of positional data for at least one spatial feature associated with a controller based on a pose of the electronic device, as determined using the depth data, relative to the at least one spatial feature associated with the controller;

receiving a set of rotational data representing three degrees-of-freedom (3DoF) orientation of the controller within the local environment; and

tracking a six degrees-of-freedom (6DoF) position of the controller within the local environment based on the set of positional data and the set of rotational data.

2. The method of claim 1, wherein the set of rotational data comprises inertial measurement unit (IMU) data.

3. The method of claim 1, further comprising:

identifying a position of a user'"'"'s hand, wherein the at least one spatial feature includes a thumb position of the user'"'"'s hand; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the thumb position.

4. The method of claim 3, further comprising:

identifying a grasp pattern of the user'"'"'s hand, wherein the at least one spatial feature further includes an identified relationship between the thumb position and the grasp pattern; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the identified relationship between the thumb position and the grasp pattern.

5. The method of claim 4, further comprising:

determining, based on the identified relationship between the thumb position and the grasp pattern, an inferred thumb position when a thumb of the user'"'"'s hand is not visible to the electronic device.

6. The method of claim 3, further comprising:

receiving a set of touchpad information from the controller, wherein the set of touchpad information includes a contact position of a thumb of the user'"'"'s hand on a touchpad of the controller.

7. The method of claim 6, further comprising:

determining an offset of the contact position from a center axis of the touchpad; and

determining the 6DoF position of the controller based on a corrected pose of the electronic device relative to the thumb position, wherein the corrected pose accounts for the offset of the contact position.

8. The method of claim 1, further comprising:

identifying a position of a visible marking on a surface of the controller; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the visible marking.

9. A system for performing controller tracking, comprising:

a depth sensor to capture depth data from a local environment proximate the electronic device;

a hand-held controller; and

a processor to receive a set of positional data based on a pose of the electronic device, as determined using the depth data, relative to at least one spatial feature associated with the hand-held controller, wherein the processor is configured to receive a set of rotational data representing 3DoF orientation of the controller within the local environment, and further wherein the processor is configured to track a 6DoF position of the controller within the local environment based on the set of positional data and the set of rotational data.

10. The system of claim 9, wherein the hand-held controller includes a set of gyroscopes and altimeters configured to capture 3DoF rotational orientation data.

11. The system of claim 9, wherein the hand-held controller includes an inertial measurement unit (IMU) configured to capture 3DoF rotational orientation data.

12. The system of claim 9, wherein the hand-held controller further includes a touchpad configured to detect a contact position of a user'"'"'s thumb.

13. The system of claim 12, wherein:

the processor is to determine an offset of the contact position from a center axis of the touchpad, and further wherein the processor is to determine the 6DoF position of the hand-held controller based on a corrected pose of the electronic device that accounts for the offset of the contact position.

14. The system of claim 9, wherein:

the processor is to determine a position of a visible marking on a surface of the controller, and further wherein the processor is to track the 6DoF position of the controller based at least on the pose of the electronic device relative to the visible marking.

15. A non-transitory computer readable medium having stored therein instructions, which when executed by a processor, perform operations comprising:

generating depth data at an electronic device based on a local environment proximate the electronic device;

generating a set of positional data based on a pose of the electronic device, as determined using the depth data, relative to at least one spatial feature associated with the controller;

receiving a set of rotational data representing three degrees-of-freedom (3DoF) orientation of the controller within the local environment; and

tracking a six degrees-of-freedom (6DoF) position of the controller within the local environment based on the set of positional data and the set of rotational data.

16. The non-transitory computer readable medium of claim 15, the instructions, which when executed by a processor, further perform operations comprising:

identifying a position of a user'"'"'s hand, wherein the at least one spatial feature includes a thumb position of the user'"'"'s hand; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the thumb position.

17. The non-transitory computer readable medium of claim 16, the instructions, which when executed by a processor, further perform operations comprising:

identifying a grasp pattern of the user'"'"'s hand, wherein the at least one spatial feature further includes an identified relationship between the thumb position and the grasp pattern; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the identified relationship between the thumb position and the grasp pattern.

18. The transitory computer readable medium of claim 16, the instructions, which when executed by a processor, further perform operations comprising:

determining, based on the identified relationship between the thumb position and the grasp pattern, an inferred thumb position when a thumb of the user'"'"'s hand is not visible to the electronic device.

19. The transitory computer readable medium of claim 16, the instructions, which when executed by a processor, further perform operations comprising:

receiving a set of touchpad information from the controller, wherein the set of touchpad information includes a contact position of a thumb of the user'"'"'s hand on a touchpad of the controller; and

determining the 6DoF position of the controller based on a corrected pose of the electronic device relative to the thumb position, wherein the corrected pose accounts for an offset of the contact position from a center axis of the touchpad.

20. The transitory computer readable medium of claim 15, the instructions, which when executed by a processor, further perform operations comprising s:

identifying a position of a visible marking on a surface of the controller; and

determining the 6DoF position of the controller based on the pose of the electronic device relative to the visible markings.