Methods and apparatus to use predicted actions in virtual reality environments

• US 9,847,079 B2
• Filed: 05/10/2016
• Issued: 12/19/2017
• Est. Priority Date: 05/10/2016
• Status: Expired due to Fees

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Clement, Manuel Christian, Welker, Stefan
• Primary Examiner(s): Warren, David
• Assistant Examiner(s): SCHREIBER, CHRISTINA MARIE
• Application Number:
  US15/151,169
• Publication Number:
  US 20170330545A1
• Time in Patent Office: 588 Days
• Field of Search: 84615
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/14 during execution voice controlled instruments G10H5/005
• CPC Class Codes: G10H 2220/131 for abstract geometric visualisation of music, e.g. for interactive editing of musical parameters linked to abstract geometric figures
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/401 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing. movement pattern or gesture sensing G10H2220/201; geolocation sensing G10H2220/355, 3D sensing with accelerometer G10H2220/395
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
• CPC Class Codes: G10H 3/00 Instruments in which the tones are generated by electromechanical means
• CPC Class Codes: G10H 7/008 Means for controlling the transition from one tone waveform to another

Abstract

Methods and apparatus to use predicted actions in VR environments are disclosed. An example method includes predicting a predicted time of a predicted virtual contact of a virtual reality controller with a virtual musical instrument, determining, based on at least one parameter of the predicted virtual contact, a characteristic of a virtual sound the musical instrument would make in response to the virtual contact, and initiating producing the sound before the predicted time of the virtual contact of the controller with the musical instrument.

Claims

1. A method comprising:

predicting a predicted time of a predicted virtual contact of a virtual reality controller with a virtual musical instrument;

determining, based on at least one parameter of the predicted virtual contact and a predicted latency, a characteristic of a virtual sound to be produced by the virtual musical instrument in response to the virtual contact; and

initiating producing the virtual sound in response to the predicted latency of the virtual contact of the virtual reality controller with the virtual musical instrument being determined.

2. The method of claim 1, wherein the parameter for the predicted contact comprises a velocity.

3. The method of claim 1, wherein the predicting the virtual contact comprises using a determined location and a determined velocity to extrapolate to a predicted future location.

4. The method of claim 3, further comprising at least one of using a captured image and/or an object tracking to determine the location and/or the velocity.

5. The method of claim 1, further comprising determining when to initiate producing the virtual sound based on a system computational load.

6. The method of claim 1, further comprising predicting the at least one parameter of the predicted virtual contact, wherein the at least one parameter comprises at least one of a velocity of impact, a location of impact, a failure to impact, a momentum, a force, a direction of impact, an area of impact, or a missed contact.

7. The method of claim 1, further comprising, when the contact does not occur, automatically adjusting a position of the virtual musical instrument so the virtual reality controller contacts the virtual musical instrument at another time.

8. The method of claim 1, further comprising:

determining a characteristic of the contact of the virtual reality controller with the virtual musical instrument; and

predicting a second virtual contact of the virtual reality controller with the virtual musical instrument based on the determining the characteristic of the contact of the virtual reality controller with the virtual musical instrument.

9. The method of claim 1, further comprising:

determining a gesture of the virtual reality controller; and

adjusting a position parameter associated with the virtual musical instrument in response to the determining the characteristic of the contact of the virtual reality controller on the virtual musical instrument.

10. The method of claim 9, wherein the position parameter comprises at least one of a location, an angle, and/or a height.

11. The method of claim 1, further comprising:

determining a gesture of the virtual reality controller; and

removing the virtual musical instrument from a virtual environment in response to the gesture.

12. The method of claim 11, wherein the second virtual contact comprises at least one of a throw, a toss, a flip, a push, a kick, or a swipe.

13. The method of claim 1, further comprising:

determining a gesture of the virtual reality controller; and

adding a second virtual musical instrument to a virtual environment in response to the gesture.

14. The method of claim 1, further comprising:

determining a gesture of the virtual reality controller; and

repositioning the virtual musical instrument in response to the gesture.

15. The method of claim 14, further comprising applying a position parameter of the repositioned virtual musical instrument to automatically position another virtual object.

16. The method of claim 15, wherein the another virtual object comprises an assistive device.

17. The method of claim 1, further comprising rendering the virtual contact of the virtual reality controller with the virtual musical instrument for display inside of a head mounted display device.

18. An apparatus comprising:

a processor; and

a non-transitory machine-readable storage media storing instruments that, when executed, causes the processor to;

predict a predicted time of a predicted virtual contact of a virtual reality controller with a virtual musical instrument;

determine, based on at least one parameter of the predicted virtual contact and a predicted latency, a characteristic of a virtual sound to be produced by the virtual musical instrument in response to the virtual contact; and

initiate producing the virtual sound in response to the predicted latency of the virtual contact of the virtual reality controller with the virtual musical instrument being determined.

19. The apparatus of claim 18, wherein the instructions, when executed, cause the processor to additionally render the virtual contact of the virtual reality controller with the virtual musical instrument for display inside of a head mounted display device.

20. A non-transitory machine-readable media storing machine-readable instructions that, when executed, cause a machine to at least:

predict a predicted time of a predicted virtual contact of a virtual reality controller with a virtual musical instrument;

determine, based on at least one parameter of the predicted virtual contact and a predicted latency, a characteristic of a virtual sound to be produced by the virtual musical instrument in response to the virtual contact; and

initiate producing of the virtual sound in response to the predicted latency of the virtual contact of the virtual reality controller with the virtual musical instrument being determined.