Mapping user interactions with a controller to a hand position

• US 10,065,111 B1
• Filed: 12/16/2015
• Issued: 09/04/2018
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: Oculus VR, LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: Patel, Nirav Rajendra, Luckey, Palmer Freeman, Hammerstein, Julian, Dycus, Chris, Higgins, Jason, Moore, David Maurice
• Primary Examiner(s): Kumar, Srilakshmi K
• Assistant Examiner(s): Castiaux, Brent D
• Application Number:
  US14/971,038
• Time in Patent Office: 993 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/214 for locating contacts on a surface, e.g. floor mats or touch pads
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/422 automatically for the purpose of assisting the player, e.g. automatic braking in a driving game
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/033 Pointing devices displaced or positioned by the user , e.g. mice, trackballs, pens or joysticks; Accessories therefor digitisers characterised by the transducing means G06F3/041
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A system maps a user'"'"'s interaction with a game controller to predefined hand positions that can be used in gaming and multimedia applications. A game controller with sensors receives user input with capacitive sensors and generates sensor outputs. A controller modeling module receives the sensor outputs and trains a model based on the outputs. The trained kinematic model maps the sensor outputs to one or more of a plurality of predefined hand positions. When an application requests the status of a user'"'"'s hand position, e.g., through an API, the mapped hand position is sent to the application.

Claims

1. A game controller comprising:

a housing adapted to a body part of a user;

an array of sensors positioned in the housing configured to generate a set of outputs from the sensors based on a proximity of a body part of a user to each of the sensors; and

a controller interface coupled to the array of sensors to receive the generated output therefrom, the controller interface configured to transmit data to a kinematic model that maps the sensor outputs from sensors to one or more hand positions of a plurality of hand positions, each of the plurality of hand positions describing a particular arrangement of the individual components of a user'"'"'s hand, the one or more hand positions useable as a user input to control a software application, wherein the kinematic model is trained by generating coefficients of one or more kinematic equations based on the sensor outputs.

2. The game controller of claim 1, wherein at least one sensor of the one or more sensors detects the location of user interaction at the position of the at least one sensor.

3. The game controller of claim 1, wherein at least one sensor of the one or more sensors detects inertial changes to at least a portion of the housing.

4. The game controller of claim 1, wherein the sensor output for a sensor of the one or more sensors varies based on user-applied pressure at the position of the sensor.

5. The game controller of claim 1, wherein the one or more arrays of one or more sensors are constructed with flexible materials.

6. The game controller of claim 1, wherein the array of sensors are positioned on a surface of the housing, within the surface of the housing, or below the surface of the housing.

7. A computer program product comprising a non-transitory computer-readable storage medium containing program code that when executed by a processor causes the processor to perform steps:

instructing a user to simulate one or more of a plurality of predefined hand positions by interacting with a game controller during the training phase, each of the predefined hand positions describing a particular arrangement of the individual components of the hand;

receiving patterns of one or more sensor outputs during the training phase from one or more sensors of the game controller, the one or more sensors configured to sense user interaction with portions of the game controller;

associating the received patterns of sensor outputs to the one or more of a plurality of predefined hand positions; and

training a kinematic model based on the associations between the received patterns of sensor outputs and the one or more of a plurality of hand positions, wherein the kinematic model is trained by generating coefficients of one or more kinematic equations based on the sensor outputs.

8. The computer program product of claim 7, wherein the steps further comprise:

receiving patterns of one or more sensor outputs during gameplay from one or more sensors on the game controller;

mapping the received patterns of sensor outputs during gameplay to one or more of a plurality of hand positions using the trained kinematic model; and

sending the mapped one or more hand positions to an application on a computing device.

9. The computer program product of claim 7, wherein the plurality of predefined hand positions include one or more hand or finger motions.

10. The computer program product of claim 7, wherein the plurality of predefined hand positions include hand positions indicating positioning of individual digits of a hand.

11. The computer program product of claim 7, wherein at least one sensor of the one or more sensors detects the location of user interaction at the position of the sensor.

12. The computer program product of claim 7, wherein at least one sensor of the one or more sensors detects inertial changes to at least a portion of the housing.

13. The computer program product of claim 7, wherein the sensor output for a sensor of the one or more sensors varies based on user-applied pressure at the position of the sensor.

14. A method comprising:

instructing a user to simulate one or more of a plurality of predefined hand positions by interacting with a game controller during the training phase, each of the predefined hand positions describing a particular arrangement of the individual components of the hand;

receiving patterns of one or more sensor outputs during the training phase from one or more sensors of the game controller, the one or more sensors configured to sense user interaction with portions of the game controller;

associating the received patterns of sensor outputs to the one or more of a plurality of predefined hand positions; and

training a kinematic model based on the associations between the received patterns of sensor outputs and the one or more of a plurality of hand positions, wherein training the kinematic model comprises;

generating coefficients of one or more kinematic equations based on the received patterns of sensor outputs.

15. The method of claim 14, wherein the steps further comprise:

receiving patterns of one or more sensor outputs during gameplay from one or more sensors on the game controller;

mapping the received patterns of sensor outputs during gameplay to one or more of a plurality of hand positions using the trained kinematic model; and

sending the mapped one or more hand positions to an application on a computing device.

16. The method of claim 14, wherein the plurality of predefined hand positions include one or more hand or finger motions.

17. The method of claim 14, wherein the plurality of predefined hand positions include hand positions indicating positioning of individual digits of a hand.

18. The method of claim 14, wherein at least one sensor of the one or more sensors detects the location of user interaction at the position of the sensor.

19. The method of claim 14, wherein at least one sensor of the one or more sensors detects inertial changes to at least a portion of the housing.

20. The method of claim 14, wherein the sensor output for a sensor of the one or more sensors varies based on user-applied pressure at the position of the sensor.