VIRTUAL REALITY HAND GESTURE GENERATION

• US 20190138107A1
• Filed: 11/19/2018
• Published: 05/09/2019
• Est. Priority Date: 10/11/2016
• Status: Active Grant

• Current Assignee: Valve Corporation
• Original Assignee: Valve Corporation
• Inventors: Nietfeld, Scott Douglas, van den Heuvel, Joe
• Granted Patent: US 10,987,573 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/218 using pressure sensors, e.g. generating a signal proportional to the pressure applied by the player
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method including receiving at least one of touch data or force data representing a touch input received at the controller, determining one or more model(s), generating image data using the one or more models, the image data representing at least a hand gesture corresponding to the touch input received at the controller, and transmitting the image data to a virtual reality (VR) environment for display.

Claims

1. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

receiving, via one or more sensors, first motion data corresponding to a first movement of a hand of a user operating one or more controllers;

receiving, via the one or more controllers, first touch data corresponding to a first touch of the hand of the user operating the one or more controllers;

receiving, via the one or more controllers, first force data corresponding to the first touch of the hand of the user operating the one or more controllers;

associating the first motion data with the first touch data and the first force data;

generating, based at least in part on the first motion data, the first touch data, and the first force data, a model corresponding to a gesture of the hand;

receiving, via the one or more sensors, second motion data corresponding to a second movement of the hand of the user operating the one or more controllers;

receiving, via the one or more controllers, second touch data corresponding to a second touch of the hand of the user operating the one or more controllers;

receiving, via the one or more controllers, second force data corresponding to the second touch of the hand of the user operating the one or more controllers;

associating the second motion data with the second touch data and the second force data;

training, based at least in part on the second motion data, the second touch data, and the second force data, the model corresponding to the gesture of the hand;

receiving, via the one or more controllers, third touch data corresponding to a third touch of the hand of the user operating the one or more controllers;

receiving, via the one or more controllers, third force data corresponding to the third touch of the hand of the user operating the one or more controllers; and

generating, based at least in part on the model corresponding to the gesture and the third force data, image data corresponding to a representation of the hand.

2. The system as recited in claim 1, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform an act comprising transmitting, to a virtual reality display, the image data corresponding to the representation of the hand.

3. The system as recited in claim 1, wherein the one or more sensors include a depth sensor.

4. The system as recited in claim 1, wherein:

the associating the first motion data with the first touch data and the first force data includes matching a first time stamp of the first motion data with a first time stamp of the first touch data and a first time stamp of the first force data; and

the associating the second motion data with the second touch data and second force data includes matching a second time stamp of the second motion data with a second time stamp of the second touch data and a second time stamp of the second force data.

5. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

receiving motion data corresponding to a movement of a hand operating a controller;

receiving touch data corresponding to touch input of the hand operating the controller;

associating the motion data and the touch data; and

generating, based at least in part on the associating the motion data and the touch data, a training model corresponding to a gesture of the hand.

6. The system as recited in claim 5, wherein the motion data comprises first motion data and the touch data comprises first touch data, the movement of the hand comprises a first movement, and the touch input of the hand comprises a first touch input, and wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

receiving second motion data corresponding to a second movement of the hand of the user operating the controller;

receiving second touch data corresponding to second touch input of the hand operating the controller; and

generating, based at least in part on the second motion data and the second touch data, an updated training model corresponding to the gesture of the hand.

7. The system as recited in claim 6, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

generating, using the updated training model, image data corresponding to a representation of the hand; and

transmitting the image data corresponding to the representation of the hand.

8. The system as recited in claim 7, wherein transmitting the image data causes a remote device to display the representation of the hand.

9. The system as recited in claim 5, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform an act comprising receiving force data corresponding to an amount of force associated with the touch input, and

wherein the generating the training model corresponding to the gesture of the hand is further based at least in part on the force data.

10. The system as recited in claim 5, wherein the touch data indicates one or more locations on the controller receiving the touch input.

11. The system as recited in claim 5, wherein associating the motion data and the touch data comprises associating a time stamp of the motion data with a time stamp of the touch data.

12. The system as recited in claim 5, wherein:

receiving the motion data comprises receiving the motion data from a camera communicatively coupled to the system; and

receiving the touch data comprises receiving the touch data from a controller communicatively coupled to the system.

13. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

receiving, from a controller, at least one of;

touch data indicating a touch input received at the controller;

or

force data indicating an amount of force associated with the touch input;

analyzing at least one of the touch data or the force with respect to a trained model that is associated with a hand gesture;

determining, based at least in part on the analyzing, that at least one of the touch data or the force data corresponds to the hand gesture;

generating image data representing the hand gesture; and

transmitting the image data for display.

14. The system as recited in claim 13, wherein the touch data indicates a location on the controller corresponding to the touch input.

15. The system as recited in claim 13, wherein the touch data comprises first touch data, the force data comprises first force data, the trained model comprises a first trained model, the touch input comprises first touch input, the image data comprises first image data, and the hand gesture comprises a first hand gesture, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

receiving, from the controller, at least one of;

second touch data representing a second touch input received at the controller;

or

second force data indicating an amount of force associated with the second touch input;

analyzing at least one of the second touch data or the second force data with respect to a second trained model that is associated with a second hand gesture;

determining that at least one of the second touch data or the second force data corresponds to a second hand gesture;

generating second image data representing the second hand gesture; and

transmitting the second image data for display.

16. The system as recited in claim 13, wherein the image data comprises first image data and the hand gesture comprises a first hand gesture, and wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

determining a second hand gesture using at least one or more predictive modeling techniques and based at least in part on the first hand gesture;

generating second image data representing the second hand gesture; and

transmitting the second image data for display.

17. The system as recited in claim 13, wherein the trained model comprises a model previously trained using at least one of:

previous touch data received from one or more controllers during a first period of time;

previous force data received from the one or more controllers during the first period of time;

or

previous image data received from one or more cameras during the first period of time.

18. The system as recited in claim 13, wherein the controller comprises a first controller, the touch data comprises first touch data, the force data comprises first force data, the image data comprises first image data, and wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

receiving, from a second controller, at least one of;

second touch data indicating a touch input received at the second controller;

or

second force data indicating an amount of force associated with the touch input received at the second controller;

analyzing at least one of the second touch data or the second force data with respect to one or more trained models that are associated with one or more hand gestures;

generating second image data representing a hand gesture received at the second controller; and

transmitting the second image data for display.

19. The system as recited in claim 13, wherein the image data comprises a three dimensional (3D) representation of the hand gesture.

20. The system as recited in claim 13, wherein the one or more non-transitory computer-readable media store computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to further perform acts comprising:

comparing at least one of the touch data or the force data to one or more trained models, wherein the one or more trained models comprise the trained model, and wherein individual trained models of the one or more trained models are associated with one or more hand gestures, and

wherein the analyzing at least one of the touch data or the force data with respect to the trained model is based at least in part on the comparing.