Holding and Releasing Virtual Objects

• US 20190325651A1
• Filed: 04/19/2019
• Published: 10/24/2019
• Est. Priority Date: 10/11/2016
• Status: Active Grant

• Current Assignee: Valve Corporation
• Original Assignee: Valve Corporation
• Inventors: Bradner, Keith, Slocum, Jeremy, Nietfeld, Scott Douglas, Yang, Lawrence, Leinbaugh, Jeffrey George
• Granted Patent: US 11,625,898 B2
• Time in Patent Office: Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/537 using indicators, e.g. showing the condition of a game character on screen
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 involving 3D image data

Abstract

Techniques and devices for holding and releasing virtual objects on a display based on input received from one or more handheld controllers are described herein. In some instances, a handheld controller includes one or more sensors, such as proximity sensors, force sensors (e.g., force resisting sensors, etc.), accelerometers, and/or other types of sensors configured to receive input from a hand of a user gripping the handheld controller. Hardware, software, and/or firmware on the controller and/or on a device coupled to the controller (e.g., a game console, a server, etc.) may receive data from these sensors and generate a representation of a corresponding gesture on a display, such as a monitor, a virtual-reality system, and/or the like.

Claims

1. A method comprising:

receiving first data from one or more sensors of a handheld controller, the first data indicating at least one of a force or proximity of at least a portion of a hand of a user holding the handheld controller at a first time;

storing, based at least in part on the first data, a first indication that a virtual object rendered on a display has been picked up by the user;

presenting, on the display and based at least in part on the first indication, a virtual hand of the user holding the virtual object;

receiving second data from the one or more sensors, the second data indicating at least one of a force or proximity of at least a portion of the hand at a second time;

storing, based at least in part on the second data, a second indication that the virtual hand is to release the virtual object within a predetermined amount of time from the second time;

receiving third data from the one or more sensors, the third data indicating a velocity of the handheld controller at a third time; and

presenting, on the display, the virtual hand releasing the virtual object prior to or upon expiration of the predetermined amount of time.

2. The method as recited in claim 1, wherein the first data indicates a force of at least a portion of the hand at the first time, and further comprising:

determining that the force at the first time is greater than a force threshold;

and wherein the storing the first indication comprises storing the first indication that the virtual object rendered on the display has been picked up by the user based at least in part on determining that the force at the first time is greater than the force threshold.

3. The method as recited in claim 1, wherein the first data comprises a first capacitance value measured by the one or more sensors, and further comprising:

determining that at least one of the first capacitance value or a second capacitance value that is based at least in part on the first capacitance value is greater than a capacitance threshold;

and wherein the storing the first indication comprises storing the first indication that the virtual object rendered on the display has been picked up by the user based at least in part on determining that the at least one of the first capacitance value or the second capacitance value is greater than the capacitance threshold.

4. The method as recited in claim 1, wherein:

the first data indicates a force of at least a portion of the hand at the first time;

the second data indicates a force of at least a portion of the hand at the second time; and

the method further comprises;

determining a difference between the force at the first time and the force at the second time; and

determining that the difference is greater than a difference threshold;

and wherein storing the second indication comprises storing the second indication that the hand is to release the virtual object based at least in part on determining that the difference is greater than the difference threshold.

5. The method as recited in claim 1, wherein the second data further indicates a velocity of the handheld controller at the second time, and further comprising:

determining that the velocity of the handheld controller at the second time is greater than a velocity threshold;

and wherein storing the second indication comprises storing the second indication that the hand is to release the virtual object based at least in part on determining that the velocity of the handheld controller at the second time is greater than the velocity threshold.

6. The method as recited in claim 1, wherein the second data comprises a first capacitance value measured by the one or more sensors, and further comprising:

determining that at least one of the first capacitance value or a second capacitance value that is based at least in part on the first capacitance value is not greater than a capacitance threshold;

and wherein storing the second indication comprises storing the second indication that the hand is to release the virtual object based at least in part on determining that the first capacitance value is not greater than the capacitance threshold.

7. The method as recited in claim 1, wherein the second data comprises a velocity of the handheld controller at the second time, and further comprising:

determining that the velocity of the handheld controller at the third time is not greater than the velocity of the handheld controller at the second time;

and wherein the presenting the virtual hand releasing the object comprises presenting, on the display, the virtual hand releasing the object based at least in part on determining that the velocity of the handheld controller at the third time is not greater than the velocity of the handheld controller at the second time.

8. The method as recited in claim 1, further comprising:

determining that the predetermined amount of time has expired without presenting the virtual hand releasing the virtual object;

and wherein the presenting the virtual hand releasing the object comprises presenting, on the display, the virtual hand releasing the object based at least in part on determining that the predetermined amount of time has expired without presenting the virtual hand releasing the virtual object.

9. The method as recited in claim 1, wherein the second data comprises a velocity of the handheld controller at the second time, and further comprising:

determining that the velocity of the handheld controller at the third time is greater than the velocity of the handheld controller at the second time;

storing an indication that the velocity of the handheld controller at the third time corresponds to a peak velocity;

storing an indication that the velocity of the handheld controller at the second time corresponds to a floor velocity;

calculating, based at least in part on the peak velocity and the floor velocity, an ending velocity; and

storing an indication of the ending velocity.

10. The method as recited in claim 9, further comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a fourth time; and

determining that the velocity of the handheld controller at the fourth time is less than the floor velocity;

and wherein the presenting the virtual hand releasing the virtual object comprises presenting the virtual hand releasing the virtual object based at least in part on determining that the velocity of the handheld controller at the fourth time is less than the floor velocity.

11. The method as recited in claim 9, further comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a fourth time;

determining that the velocity of the handheld controller at the fourth time is not less than the floor velocity;

determining that the velocity of the handheld controller at the fourth time is less than the ending velocity; and

determining that a velocity of the handheld controller has remained less than the ending velocity for greater than a threshold amount of time;

and wherein the presenting the virtual hand releasing the virtual object comprises presenting the virtual hand releasing the virtual object based at least in part on determining that the velocity of the handheld controller has remained less than the ending velocity for greater than the threshold amount of time.

12. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

receiving first data from one or more sensors of a handheld controller, the first data indicating a force or proximity of at least a portion of a hand of a user holding the handheld controller;

storing an indication that a virtual object is to be held based at least in part on the first data;

receiving second data from the one or more sensors, the second data comprising a velocity of the handheld controller; and

storing an indication that the virtual object is to be released based at least in part on the second data.

13. The system as recited in claim 12, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

determining that the velocity of the handheld controller is less than a previously measured velocity of the handheld controller;

and wherein the storing the indication that the virtual object is to be released comprises storing the indication that the virtual object is to be released based at least in part on determining that the velocity of the handheld controller is less than the previously measured velocity of the handheld controller.

14. The system as recited in claim 12, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

determining that a predetermined amount of time has elapsed since storing the indication that the virtual object is to be released; and

causing a display to present release of the virtual object based at least in part on determining that the predetermined amount of time has elapsed.

15. The system as recited in claim 12, wherein the velocity of the handheld controller comprises a velocity of the handheld controller at a first time, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving third data from the one or more sensors, the third data comprising a velocity of the handheld controller at a second time that is after the first time;

determining that the velocity of the handheld controller at the second time is not greater than the velocity of the handheld controller at the first time; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller at the second time is not greater than the velocity of the handheld controller at the first time.

16. The system as recited in claim 12, wherein the velocity of the handheld controller comprises a velocity of the handheld controller at a first time, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving third data from the one or more sensors, the third data comprising a velocity of the handheld controller at a second time that is after the first time;

determining that the velocity of the handheld controller at the second time is greater than the velocity of the handheld controller at the first time;

storing an indication that the velocity of the handheld controller at the second time corresponds to a peak velocity;

storing an indication that the velocity of the handheld controller at the first time corresponds to a floor velocity;

calculating an ending velocity based at least in part on at least one the peak velocity and the floor velocity; and

storing an indication of the ending velocity.

17. The system as recited in claim 16, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a third time;

determining that the velocity of the handheld controller at the third time is less than the floor velocity; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller at the third time is less than the floor velocity.

18. The system as recited in claim 16, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a third time;

determining that the velocity of the handheld controller at the third time is not less than the floor velocity;

determining that the velocity of the handheld controller at the third time is less than the ending velocity;

determining that a velocity of the handheld controller has remained less than the ending velocity for greater than a threshold amount of time; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller has remained less than the ending velocity for greater than the threshold amount of time.

19. A handheld controller comprising:

a controller body;

one or more sensors coupled to the controller body;

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

receiving first data from the one or more sensors, the first data indicating a force or proximity of at least a portion of a hand of a user holding the handheld controller;

storing an indication that a virtual object is to be held based at least in part on the first data;

receiving second data from the one or more sensors, the second data comprising a velocity of the handheld controller; and

storing an indication that the virtual object is to be released based at least in part on the second data.

20. The handheld controller as recited in claim 19, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

determining that the velocity of the handheld controller is less than a previously measured velocity of the handheld controller;

and wherein the storing the indication that the virtual object is to be released comprises storing the indication that the virtual object is to be release based at least in part on determining that the velocity of the handheld controller is less than the previously measured velocity of the handheld controller.

21. The handheld controller as recited in claim 19, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

determining that a predetermined amount of time has elapsed since storing the indication that the virtual object is to be released; and

causing a display to present release of the virtual object based at least in part on determining that the predetermined amount of time has elapsed.

22. The handheld controller as recited in claim 19, wherein the velocity of the handheld controller comprises a velocity of the handheld controller at a first time, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving third data from the one or more sensors, the third data comprising a velocity of the handheld controller at a second time that is after the first time;

determining that the velocity of the handheld controller at the second time is not greater than the velocity of the handheld controller at the first time; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller at the second time is not greater than the velocity of the handheld controller at the first time.

23. The handheld controller as recited in claim 19, wherein the velocity of the handheld controller comprises a velocity of the handheld controller at a first time, and the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving third data from the one or more sensors, the third data comprising a velocity of the handheld controller at a second time that is after the first time;

determining that the velocity of the handheld controller at the second time is greater than the velocity of the handheld controller at the first time;

storing an indication that the velocity of the handheld controller at the second time corresponds to a peak velocity;

storing an indication that the velocity of the handheld controller at the first time corresponds to a floor velocity;

calculating an ending velocity based at least in part on the peak velocity and the floor velocity; and

storing an indication of the ending velocity.

24. The handheld controller as recited in claim 23, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a third time;

determining that the velocity of the handheld controller at the third time is less than the floor velocity; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller at the third time is less than the floor velocity.

25. The handheld controller as recited in claim 23, wherein the one or more computer-readable media further store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising:

receiving fourth data from the one or more sensors, the fourth data indicating a velocity of the handheld controller at a third time;

determining that the velocity of the handheld controller at the third time is not greater than the velocity of the handheld controller at the second time;

determining that the velocity of the handheld controller at the third time is not less than the floor velocity;

determining that the velocity of the handheld controller at the third time is less than the ending velocity;

determining that a velocity of the handheld controller has remained less than the ending velocity for greater than a threshold amount of time; and

causing a display to present release of the virtual object based at least in part on determining that the velocity of the handheld controller has remained less than the ending velocity for greater than the threshold amount of time.

26. The handheld controller as recited in claim 19, wherein the one or more sensors comprise:

an accelerometer configured to determine the velocity of the handheld controller;

a force sensing resistor (FSR) configured to generate data indicating force of at least a portion of the hand of the user on the handheld controller; and

a proximity sensor configured to generate data indicating proximity of at least a portion of the hand of the user on the handheld controller.