Electronic controller with finger motion sensing

• US 10,874,939 B2
• Filed: 06/15/2018
• Issued: 12/29/2020
• Est. Priority Date: 06/16/2017
• Status: Active Grant

• Current Assignee: Valve Corporation
• Original Assignee: Valve Corporation
• Inventors: Nietfeld, Scott, Leinbaugh, Jeffrey George, Hope, Eric James, Bryan, Philip, Britt, Scott Richard, Mucha, Jeffrey Walter, Gottschalk, Claire Michelle, Slocum, Jeremy
• Primary Examiner(s): Lim, Seng H
• Application Number:
  US16/010,385
• Publication Number:
  US 20180361234A1
• Time in Patent Office: 928 Days
• Field of Search: None
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/214 for locating contacts on a surface, e.g. floor mats or touch pads
• CPC Class Codes: A63F 13/245 specially adapted to a particular type of game, e.g. steering wheels
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: B25J 13/02 Hand grip control means handles or pedals for crane control B66C13/56; for measuring the force applied to control members G01L5/22; hand-held casings for switching devices, e.g. joy-sticks H01H9/0214
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A controller includes a body having a handle, and an array of proximity sensors spatially distributed on, in, beneath, or near the outer surface of the handle, responsive to a proximity of a user'"'"'s fingers to that outer surface. A finger tracker converts the output of the array of proximity sensors to a set of joint angles corresponding to a plurality of the user'"'"'s fingers. The controller may include a renderer for processing the joint angles to deform a hand mesh that is rendered for display. Values may be calculated to facilitate normalization of the output of the proximity sensor array and thereby generate a set of normalized finger detection data. This data may be processed through curl logic to produce a linear estimate of gross finger curl with respect to the user and thereby generate a set of estimates for a plurality of finger joint angles for the user.

Claims

1. A system comprising:

a handheld controller comprising;

a controller body having a handle; and

a proximity sensor coupled to the handle and responsive to a proximity of a finger of a user to the handle;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving, from the proximity sensor, proximity data representing the proximity of the finger of the user to the handle;

determining an unfiltered finger curl value based at least in part on the proximity data;

passing the unfiltered finger curl value through one or more filters to obtain a filtered finger curl value; and

determining an estimated joint angle of the finger based at least in part on the filtered finger curl value.

2. The system of claim 1, the operations further comprising:

processing the estimated joint angle to deform a hand mesh that is rendered for display.

3. The system of claim 1, wherein the determining of the unfiltered finger curl value based at least in part on the proximity data comprises:

calculating a set of values based at least in part on the proximity data;

generating, based at least in part on the set of values, a set of normalized finger detection data; and

processing the set of normalized finger detection data to determine the unfiltered finger curl value.

4. The system of claim 1, wherein the handheld controller is for use in conjunction with a virtual reality system.

5. The system of claim 1, wherein the handheld controller includes a tracking arc that is fixed to the controller body.

6. The system of claim 5, wherein the tracking arc includes a first plurality of tracking sensors, the first plurality of tracking sensors being responsive to electromagnetic radiation emitted by an electronic device of the system.

7. The system of claim 1, wherein the handheld controller includes a hand retainer configured to physically bias a palm of the user against an outside surface of the handle.

8. The system of claim 1, wherein the controller body further comprises a distal head that extends from the handle, the distal head including:

a distal end, and

at least one thumb-operated control disposed between the handle and the distal end.

9. The system of claim 1, wherein the proximity sensor comprises a plurality of capacitive sensors embedded under an outer surface of the handle, the outer surface comprising an electrically insulative material.

14. The system of claim 1, wherein the proximity sensor is incorporated into one or more flexible printed circuits, and wherein each of the one or more flexible printed circuits is attached to an inner surface of the handle.

15. The system of claim 1, wherein the proximity sensor comprises an array comprising a plurality of capacitive sensors.

19. The system of claim 1, wherein the estimated joint angle of the finger comprises a first estimated joint angle of the finger, the operations further comprising:

determining a joint model associated with the finger of the user; and

determining, based at least in part on the joint model and the first estimated joint angle, a second estimated joint angle of the finger.

20. The system of claim 1, wherein the finger of the user comprises a first finger of the user, the handheld controller further comprising:

an additional proximity sensor coupled to the handle and responsive to a proximity of a second finger of the user to the handle;

and

wherein the operations further comprise;

receiving, from the additional proximity sensor, additional proximity data representing the proximity of the second finger of the user to the handle, the additional proximity data being used to determine the estimated joint angle of the first finger.

21. The system of claim 1, wherein the finger of the user comprises a first finger of the user, the handheld controller further comprising:

a force-sensing resistor (FSR) coupled to the handle and responsive to at least one of the first finger or a second finger of the user grasping the handle; and

wherein the operations further comprise;

receiving, from the FSR, force data representing a force of the at least one of the first finger or the second finger of the user grasping the handle, the force data being used to determine the estimated joint angle of the first finger.

27. The system of claim 1, wherein the passing of the unfiltered finger curl value through the one or more filters comprises passing the unfiltered finger curl value through:

a median filter to reject transient spikes; and

a backlash filter to remove baseline sensor noise.

10. A method comprising:

receiving first data from a proximity sensor that represents a proximity of a finger of a user relative to a handle of a controller;

determining an unfiltered finger curl value based at least in part on the first data;

passing the unfiltered finger curl value through one or more filters to obtain a filtered finger curl value; and

determining an estimate of a joint angle of the finger based at least in part on the filtered finger curl value.

11. The method according to claim 10, further comprising:

generating a hand mesh based at least in part on the estimate of the joint angle of the finger; and

rendering the hand mesh for display.

12. The method according to claim 10, further comprising:

receiving second data from the proximity sensor prior to the receiving of the first data;

determining that the second data corresponds to a predetermined hand gesture of the user; and

initializing finger tracking in response to the determining that the second data corresponds to the predetermined hand gesture.

13. The method according to claim 12, wherein the predetermined hand gesture comprises an ungrasp gesture.

16. The method according to claim 10, further comprising:

calculating a plurality of values based at least in part on the first data to facilitate normalization of the proximity sensor with respect to the user; and

generating a set of normalized finger detection data based at least in part on the plurality of values.

17. The method according to claim 16, further comprising:

processing the set of normalized finger detection data through a set of curl logic to produce an estimate of finger curl with respect to the user; and

calculating the estimate of the joint angle based at least in part on the estimate of the finger curl.

18. The method according to claim 17, wherein the estimate of the joint angle is a linear estimate.

22. The method according to claim 10, wherein the finger of the user comprises a first finger of the user, the method further comprising:

receiving second data from an additional proximity sensor that represents a proximity of a second finger of the user relative to the handle, wherein the second data is used to determine the estimate of the joint angle of the first finger.

23. The method according to claim 10, wherein the finger of the user comprises a first finger of the user, the method further comprising:

receiving second data from a force-sensing resistor (FSR) that represents a force of at least one of the first finger or a second finger of the user grasping the handle, wherein the second data is used to determine the estimate of the joint angle of the first finger.

24. A device comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

receiving proximity data generated by a proximity sensor, the proximity data representing a proximity of a finger of a user to a handle of a controller;

determining an unfiltered finger curl value based at least in part on the proximity data;

passing the unfiltered finger curl value through one or more filters to obtain a filtered finger curl value; and

determining an estimated joint angle of the finger based at least in part on the filtered finger curl value.

25. The device of claim 24, the operations further comprising:

receiving additional proximity data generated by an additional proximity sensor, the additional proximity data representing a proximity of a second finger of the user relative to the handle of the controller, and

wherein determining the estimated joint angle of the finger is further based at least in part on the additional proximity data.

26. The device of claim 24, the operations further comprising:

receiving force data generated by a force-sensing resistor (FSR), the force data representing a force of at least one of the finger or a second finger of the user grasping the handle, and

wherein determining the estimated joint angle of the finger is further based at least in part on the force data.