Gesture-based user interface for navigating a menu

• US 10,357,714 B2
• Filed: 10/27/2010
• Issued: 07/23/2019
• Est. Priority Date: 10/27/2009
• Status: Active Grant

• Current Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Original Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Inventors: Challinor, Ryan, Matheson, Dare, Crooker, Kasson D., Flury, Marc A., Boch, Matthew C., Egozy, Eran B., Tate, Dean N.
• Primary Examiner(s): Chow, Doon Y
• Assistant Examiner(s): Nguyen, Le V
• Application Number:
  US12/913,493
• Publication Number:
  US 20110185309A1
• Time in Patent Office: 3,191 Days
• Field of Search: 715863, 715822, 715784
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/814 Musical performances, e.g. by evaluating the player's ability to follow a notation
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

The invention includes methods, systems, computer program products and means for providing an intuitive user interface for interacting with a game console that obviates the need for, or use of, a typical game controller. In one embodiment, the user interface comprises an analog-like mechanism for manipulating a menu and icon system. The menu system allows the user to highlight a menu option or icon using movement along one axis and to activate the menu option or icon via movement on a different axis. In one embodiment, highlighting is associated with a vertical motion and activation is associated with a horizontal motion. The vertical motion can be tied to the vertical movement of a particular appendage, e.g., the up and down movement of the user'"'"'s right arm, and the horizontal motion can be tied to the horizontal movement of that or another appendage, e.g., movement of the right arm or hand across the body from right to left.

Claims

1. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

displaying a menu on the display comprising menu items displaced axially along a display axis having a first direction;

receiving a 3D skeleton having a joint associated with a corresponding reference point on a player'"'"'s body;

determining a position for the joint along a linear gesture axis that is parallel to the first direction, where the joint is associated with a highlighting action;

highlighting a menu item based on the position of the joint along the linear gesture axis;

determining a plurality of vectors, wherein each vector represents a movement of the joint defined by a difference in position of the joint between a pair of two consecutive frames;

identifying at least some vectors within the plurality of vectors that satisfy one or more conditions, wherein the one or more conditions include a condition that the at least some vectors each have a length greater than a predetermined minimum movement threshold;

summing at least a component of the at least some vectors, while not summing other vectors in the plurality of vectors that do not satisfy the one or more conditions;

determining that the sum of the at least a component of the at least some vectors is greater than a specified length; and

activating the highlighted menu item in response to the determination that the sum is greater than the specified length.

2. The method of claim 1, further comprising assigning a set of position values to each menu item.

3. The method of claim 1, wherein the one or more conditions further include a condition that the at least some vectors are each within an angle threshold of perpendicular to the first direction.

4. The method of claim 1, wherein the highlighted menu item is activated only when the joint moves less than a certain distance in the first direction.

5. The method of claim 1, further comprising:

determining that the joint has moved a minimum distance that is less than the specified length along a second direction perpendicular to the first direction;

in response to the determination that the joint has moved the minimum distance along the second direction, locking the highlighted menu item so that changes in the position of the joint along the linear gesture axis do not change the highlighted menu item.

6. The method of claim 1, further comprising:

assigning a position value to a scroll area;

determining that the scroll area is active based on the position of the joint and the position value of the scroll area; and

when the scroll area is active, scrolling at least one different menu item onto the display.

7. The method of claim 1, further comprising:

determining that the joint has moved a minimum distance that is less than the specified length along a second direction perpendicular to the first direction;

in response to the determination that the joint has moved the minimum distance along the second direction, altering the appearance of the highlighted menu item to indicate a beginning of an activation action.

8. The method of claim 1, wherein at least some of the at least some vectors that satisfy the one or more conditions are associated with non-consecutive pairs of frames.

9. A processor-based system for interacting with a menu based on input received via a sensor system, the system comprising:

the sensor system configured to sense input from a player, the sensor system in communication with a game platform having a processor;

a display in communication with the game platform;

the processor in the game platform being configured to;

display a menu on the display comprising menu items displaced axially along a display axis having a first direction;

receive a 3D skeleton having a joint associated with a corresponding reference point on a player'"'"'s body;

determine a position for the joint along a linear gesture axis that is parallel to the first direction, where the joint is associated with a highlighting action;

highlight a menu item based on the position of the joint along the linear gesture axis;

determine a plurality of vectors, wherein each vector represents a movement of the joint defined by a difference in position of the joint between a pair of two consecutive frames;

identify at least some vectors within the plurality of vectors that satisfy one or more conditions, wherein the one or more conditions include a condition that the at least some vectors each have a length greater than a predetermined minimum movement threshold;

sum at least a component of the at least some vectors, while not summing other vectors in the plurality of vectors that do not satisfy the one or more conditions;

determine that the sum of the at least a component of the at least some vectors is greater than a specified length; and

activate the highlighted menu item in response to the determination that the sum is greater than the specified length.

10. The system of claim 9, wherein the processor is further configured to assign a set of position values to each menu item.

11. The system of claim 9, wherein the processor is further configured to:

determine that the joint has moved a minimum distance that is less than the specified length along a second direction perpendicular to the first direction;

in response to the determination that the joint has moved the minimum distance along the second direction, alter the appearance of the highlighted menu item to indicate a beginning of an activation action.

12. The system of claim 9, wherein the one or more conditions further include a condition that the at least some vectors are each within an angle threshold of perpendicular to the first direction.

13. The system of claim 9, wherein the processor is configured to activate the highlighted menu item only when the joint moves less than a certain distance in the first direction.

14. The system of claim 9, wherein the processor is further configured to:

determine that the joint has moved a minimum distance that is less than the specified length along a second direction perpendicular to the first direction;

in response to the determination that the joint has moved the minimum distance along the second direction, lock the highlighted menu item so that changes in the position of the joint along the first direction do not change the highlighted menu item.

15. The system of claim 9, wherein the processor is further configured to:

assign a position value to a scroll area;

determine that the scroll area is active based on the position of the joint and the position value of the scroll area; and

when the scroll area is active, scroll at least one different menu item onto the display.

16. The system of claim 9, wherein at least some of the at least some vectors that satisfy the one or more conditions are associated with non-consecutive pairs of frames.