Gesture-based music game

• US 10,220,303 B1
• Filed: 03/15/2014
• Issued: 03/05/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Original Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Inventors: Schmidt, Daniel A., Boch, Matthew C., Booth, Jason Deling, Gibson, Brian Hodges, Toepel, James Gerard, Mandel, Michael, Mintz, Jonathan, Sullivan, Mark
• Primary Examiner(s): Liddle, Jay
• Assistant Examiner(s): Rada, II, Alex F. R. P.
• Application Number:
  US14/214,827
• Time in Patent Office: 1,816 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/814 Musical performances, e.g. by evaluating the player's ability to follow a notation

Abstract

A method including displaying a multi-part visual cue that instructs a player to perform a gesture at a specified time, the multi-part visual cue including a first part indicating the gesture that is to be performed at the specified time, and a second part providing an indication of i) the specified time, and ii) a preparation period before the specified time, wherein the distance between the first and the second parts is variable over time, receiving, from a video camera, position information associated with positions of the player over time, determining a first displacement of the player using the position information, determining whether the first displacement of the player matches a first target displacement criterion associated with the multi-part visual cue, when the first displacement matches the first target displacement criterion within a timing window of the specified time, altering a gameplay characteristic of the video game.

Claims

1. A computerized method for use in a video game, the method comprising:

displaying, on a display, a multi-part visual cue that instructs a player to perform a target gesture at a specified time, the multi-part visual cue including;

a first part indicating the target gesture that is to be performed by the player at the specified time; and

a second part providing an indication of i) the specified time, and ii) a preparation period before the specified time, wherein the distance between the first and the second parts is variable over time;

receiving, from a video camera that captures images of at least part of the player'"'"'s body, position information associated with a plurality of positions of the at least part of the player'"'"'s body over a time period;

determining a first displacement of the at least part of the player'"'"'s body using the position information based on at least two of the plurality of positions, wherein the at least two of the plurality of positions each correspond to a different time within the time period;

determining whether the first displacement of the at least part of the player'"'"'s body matches a first target displacement criterion associated with the multi-part visual cue at least in part by determining a first instantaneous error score based on an angle between the first displacement and a first direction;

when the first displacement matches the first target displacement criterion within a timing window of the specified time, altering a first gameplay characteristic of the video game;

determining a second displacement of the at least part of the player'"'"'s body using the position information from the video camera;

determining whether the second displacement matches a second target displacement criterion associated with the multi-part visual cue at least in part by determining a second instantaneous error score based on an angle between the second displacement and a second direction; and

when the second displacement matches the second target displacement criterion, altering a second gameplay characteristic of the video game based on a sum of the first instantaneous error score and the second instantaneous error score.

2. The computerized method of claim 1 wherein:

the first part of the multi-part visual cue indicates that the at least part of the player'"'"'s body is to remain motionless; and

the second part of the multi-part visual cue indicates how long the at least part of the player'"'"'s body is to remain motionless.

3. The computerized method of claim 1 wherein the first target displacement criterion is one of:

a magnitude of the displacement of the at least part of the player'"'"'s body and a direction of the displacement of the at least part of the player'"'"'s body.

4. The computerized method of claim 1 wherein the first target displacement criterion requires the at least part of the player'"'"'s body to displace in one of a horizontal direction, a vertical direction, and a direction toward the video camera.

5. The computerized method of claim 1 further comprising:

displaying on the display a third part of the multi-part cue indicating an alternative gesture that can be performed by the player at the specified time;

determining whether the first displacement matches an alternative target displacement criterion associated with the alternative gesture; and

when the first displacement matches the alternative target displacement criterion within the timing window of the specified time, altering a gameplay characteristic of the video game.

6. The computerized method of claim 1 further comprising:

displaying, on the display, an additional multi-part visual cue that instructs an additional player to perform an additional gesture at an additional specified time;

receiving, from the video camera, additional position information associated with positions of at least part of the additional player over time;

determining a first displacement of the at least part of the additional player using the additional position information;

determining whether the first displacement of the at least part of the additional player matches second target displacement criterion associated with the additional multi-part visual cue; and

when the first displacement of the at least part of the additional player matches the second target displacement criterion within a timing window of the additional specified time, altering a gameplay characteristic of the video game.

7. The computerized method of claim 1 wherein the specified time and the additional specified time are the same.

8. The computerized method of claim 1 further comprising:

displaying, on the display, a primary cursor that is controlled by the player via the video camera;

detecting an interaction between the primary cursor and an object displayed on the display;

in response to detecting the interaction;

constraining the primary cursor to the object; and

displaying a secondary cursor that is controlled by the player via the video camera system to manipulate the object.

9. A video game system comprising:

a memory storing computer executable instructions; and

one or more processors coupled to the memory and configured to execute the instructions such that the one or more processors;

cause a display of a multi-part visual cue that instructs a player to perform a target gesture at a specified time, the multi-part visual cue including;

a first part indicating the target gesture that is to be performed by the player at the specified time; and

a second part providing an indication of i) the specified time, and ii) a preparation period before the specified time, wherein the distance between the first and the second parts is variable over time;

receive, from a video camera that captures images of at least part of the player'"'"'s body, position information associated with a plurality of positions of the at least part of the player'"'"'s body over a time period;

determine a first displacement of the at least part of the player'"'"'s body using the position information based on at least two of the plurality of positions, wherein the at least two of the plurality of positions each correspond to a different time within the time period;

determine whether the first displacement of the at least part of the player'"'"'s body matches a first target displacement criterion associated with the multi-part visual cue at least in part by determining a first instantaneous error score based on an angle between the first displacement and a first direction;

when the first displacement matches the first target displacement criterion within a timing window of the specified time, alter a first gameplay characteristic of the video game;

determine a second displacement of the at least part of the player'"'"'s body using the position information from the video camera;

determine whether the second displacement matches a second target displacement criterion associated with the multi-part visual cue at least in part by determining a second instantaneous error score based on an angle between the second displacement and a second direction; and

when the second displacement matches the second target displacement criterion, alter a second gameplay characteristic of the video game based on a sum of the first instantaneous error score and the second instantaneous error score.

10. The video game system of claim 9 wherein:

the first part of the multi-part visual cue indicates that the at least part of the player'"'"'s body is to remain motionless; and

the second part of the multi-part visual cue indicates how long the at least part of the player'"'"'s body is to remain motionless.

11. The video game system of claim 9 wherein the first target displacement criterion is one of:

a magnitude of the displacement of the at least part of the player'"'"'s body and a direction of the displacement of the at least part of the player'"'"'s body.

12. The video game system of claim 9 wherein the first target displacement criterion requires the at least part of the player'"'"'s body to displace in one of a horizontal direction, a vertical direction, and a direction toward the video camera.

13. The video game system of claim 9 wherein the computer executable instructions are further configured to cause the one or more processors to:

cause the display of a third part of the multi-part cue indicating an alternative gesture that can be performed by the player at the specified time; and

determine whether the first displacement matches an alternative target displacement criterion associated with the alternative gesture; and

when the first displacement matches the alternative target displacement criterion within the timing window of the specified time, alter a gameplay characteristic of the video game.

14. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to:

display, on a display, a multi-part visual cue that instructs a player to perform a target gesture at a specified time, the multi-part visual cue including;

a first part indicating the target gesture that is to be performed by the player at the specified time; and

a second part providing an indication of i) the specified time, and ii) a preparation period before the specified time, wherein the distance between the first and the second parts is variable over time;

receive, from a video camera that captures images of at least part of the player'"'"'s body, position information associated with a plurality of positions of the at least part of the player'"'"'s body over a time period;

determine a first displacement of the at least part of the player'"'"'s body using the position information based on at least two of the plurality of positions, wherein the at least two of the plurality of positions each correspond to a different time within the time period;

determine whether the first displacement of the at least part of the player'"'"'s body matches a first target displacement criterion associated with the multi-part visual cue at least in part by determining a first instantaneous error score based on an angle between the first displacement and a first direction;

when the first displacement matches the first target displacement criterion within a timing window of the specified time, alter a first gameplay characteristic of the video game;

determine a second displacement of the at least part of the player'"'"'s body using the position information from the video camera;

determine whether the second displacement matches a second target displacement criterion associated with the multi-part visual cue at least in part by determining a second instantaneous error score based on an angle between the second displacement and a second direction; and

when the second displacement matches the second target displacement criterion, alter a second gameplay characteristic of the video game based on a sum of the first instantaneous error score and the second instantaneous error score.

15. The non-transitory computer readable medium of claim 14 wherein:

the first part of the multi-part visual cue indicates that the at least part of the player'"'"'s body is to remain motionless; and

the second part of the multi-part visual cue indicates how long the at least part of the player'"'"'s body is to remain motionless.

16. The non-transitory computer readable medium of claim 14 wherein the first target displacement criterion is one of:

a magnitude of the displacement of the at least part of the player'"'"'s body and a direction of the displacement of the at least part of the player'"'"'s body.

17. The non-transitory computer readable medium of claim 14 wherein the first target displacement criterion requires the at least part of the player'"'"'s body to displace in one of a horizontal direction, a vertical direction, and a direction toward the video camera.

18. The non-transitory computer readable medium of claim 14 wherein the instructions are further configured to cause the one or more processors to:

display on the display a third part of the multi-part cue indicating an alternative gesture that can be performed by the player at the specified time;

determine whether the first displacement matches an alternative target displacement criterion associated with the alternative gesture; and

when the first displacement matches the alternative target displacement criterion within the timing window of the specified time, alter a gameplay characteristic of the video game.