Methods and apparatus to use predicted actions in virtual reality environments

• US 10,573,288 B2
• Filed: 12/07/2017
• Issued: 02/25/2020
• Est. Priority Date: 05/10/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Clement, Manuel Christian, Welker, Stefan
• Primary Examiner(s): Warren, David S
• Assistant Examiner(s): Schreiber, Christina M
• Application Number:
  US15/834,540
• Publication Number:
  US 20180108334A1
• Time in Patent Office: 810 Days
• Field of Search: 84615
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/14 during execution voice controlled instruments G10H5/005
• CPC Class Codes: G10H 2220/131 for abstract geometric visualisation of music, e.g. for interactive editing of musical parameters linked to abstract geometric figures
• CPC Class Codes: G10H 2220/201 for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
• CPC Class Codes: G10H 2220/401 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing. movement pattern or gesture sensing G10H2220/201; geolocation sensing G10H2220/355, 3D sensing with accelerometer G10H2220/395
• CPC Class Codes: G10H 2220/455 Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
• CPC Class Codes: G10H 3/00 Instruments in which the tones are generated by electromechanical means
• CPC Class Codes: G10H 7/008 Means for controlling the transition from one tone waveform to another

Abstract

Methods and apparatus to use predicted actions in VR environments are disclosed. An example method includes predicting a predicted time of a predicted virtual contact of a virtual reality controller with a virtual object, determining, based on at least one parameter of the predicted virtual contact, a characteristic of a virtual output the object would make in response to the virtual contact, and initiating producing the virtual output before the predicted time of the virtual contact of the controller with the virtual object.

Claims

1. A method comprising:

predicting a predicted time of a predicted virtual contact of a virtual reality controller with a virtual object within a virtual environment displayed by head-mounted device;

determining, based on at least one parameter of the predicted virtual contact and a predicted latency, a characteristic of a virtual output to be produced by the virtual object in response to the virtual contact; and

initiating producing the virtual output in response to the predicted latency of the virtual contact of the virtual reality controller with the virtual object being determined.

2. The method of claim 1, wherein the virtual object is at least one of a musical instrument, a document, a household item, a door knob, or a table.

3. The method of claim 1, wherein the virtual output is at least one of sound, light, color of light, color saturation, or acoustic shape of a sound.

4. The method of claim 1, wherein the predicting the virtual contact is predicted using a determined location and a determined velocity to extrapolate to a predicted future location.

5. The method of claim 1, further comprising predicting the at least one parameter of the predicted virtual contact, wherein the at least one parameter comprises at least one of a velocity of impact, a location of impact, a failure to impact, a momentum, a force, a direction of impact, an area of impact, or a missed contact.

6. The method of claim 1, further comprising, when the contact does not occur, automatically adjusting a position of the virtual object so the virtual reality controller contacts the virtual object at another time.

7. The method of claim 1, further comprising:

determining a characteristic of the virtual contact of the virtual reality controller with the virtual object, the virtual contact being a first virtual contact; and

predicting a second virtual contact of the virtual reality controller with the virtual object based on the determining the characteristic of the first virtual contact of the virtual reality controller with the virtual object.

8. The method of claim 7, further comprising:

determining a gesture of the virtual reality controller; and

adjusting a position parameter associated with the virtual object in response to the determining the characteristic of the virtual contact of the virtual reality controller on the virtual object.

9. The method of claim 8, wherein the position parameter comprises at least one of a location, an angle, or a height.

10. The method of claim 8, wherein the gesture includes at least one of a throw, a toss, a flip, a push, a kick, or a swipe.

11. The method of claim 1, further comprising:

determining a gesture of the virtual reality controller; and

repositioning the virtual object in response to the gesture.

12. The method of claim 11, further comprising applying a position parameter of the repositioned virtual object to automatically position another virtual object.

13. An apparatus comprising:

a processor; and

a non-transitory machine-readable storage media storing instruments that, when executed, causes the processor to;

determine a current location, a current direction and a current velocity of a virtual reality controller with respect to a virtual object within a virtual environment displayed by a head-mounted device;

predict a predicted time of a predicted virtual contact of the virtual reality controller with the virtual object;

initiate producing a virtual output based on the predicted time of the virtual contact of the virtual reality controller with the virtual object; and

determine a predicted future location based on at least the current location, the current direction and the current velocity.

14. The apparatus of claim 13, wherein the virtual object is at least one of a musical instrument, a document, a household item, a door knob, or a table.

15. The apparatus of claim 13, wherein the virtual output is at least one of sound, light, color of light, color saturation, or acoustic shape of a sound.

16. The apparatus of claim 13, further comprising tracking a predicted latency from when the virtual output is initiated and when the object output is started to be rendered.

17. The apparatus of claim 16, wherein the predicted latency is determined from at least one of an average, a windowed average, a moving average, or an exponential average.

18. A non-transitory machine-readable media storing machine-readable instructions that, when executed, cause a machine to at least:

predict a predicted time of a predicted virtual contact of a virtual reality controller with a virtual object within a virtual environment displayed by a head-mounted device;

determine, based on at least one parameter of the predicted virtual contact and a predicted latency, a characteristic of a virtual output to be produced by the virtual object in response to the virtual contact; and

initiate producing the virtual output in response to the predicted latency of the virtual contact of the virtual reality controller with the virtual object being determined.

19. The non-transitory media of claim 18, wherein the predicted virtual contact is predicted using the at least one parameter to determine a predicted future location.

20. The non-transitory media of claim 19, wherein the at least one parameter comprises at least one of a velocity of impact or a location of impact.