Gesture-based user interface

• US 10,421,013 B2
• Filed: 05/29/2013
• Issued: 09/24/2019
• Est. Priority Date: 10/27/2009
• Status: Active Grant

• Current Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Original Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Inventors: Challinor, Ryan, Matheson, Dare, Crooker, Kasson D., Flury, Marc A., Boch, Matthew C., Egozy, Eran B., Tate, Dean N.
• Primary Examiner(s): Sax, Steven P
• Assistant Examiner(s): Nguyen, Le V
• Application Number:
  US13/904,835
• Publication Number:
  US 20130260884A1
• Time in Patent Office: 2,309 Days
• Field of Search: 715863, 715822, 715784
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/814 Musical performances, e.g. by evaluating the player's ability to follow a notation
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

The invention includes methods, systems, computer program products and means for providing an intuitive user interface for interacting with a game console that obviates the need for, or use of, a typical game controller. In one embodiment, the user interface comprises an analog-like mechanism for manipulating a menu and icon system. The menu system allows the user to highlight a menu option or icon using movement along one axis and to activate the menu option or icon via movement on a different axis. In one embodiment, highlighting is associated with a vertical motion and activation is associated with a horizontal motion. The vertical motion can be tied to the vertical movement of a particular appendage, e.g., the up and down movement of the user'"'"'s right arm, and the horizontal motion can be tied to the horizontal movement of that or another appendage, e.g., movement of the right arm or hand across the body from right to left.

Claims

1. A non-transitory computer readable medium storing instructions that, when executed by a computer, cause the computer to:

display on a display a plurality of menu items displaced axially along a display axis;

receive data from a camera system, the data including a representation of a body of a user;

determine a first reference point associated with a first location on the player'"'"'s body based on the data received from the camera system;

determine a scale vector between a second reference point associated with a second location on the player'"'"'s body and a third reference point associated with a third location on the player'"'"'s body, the first, second, and third locations on the player'"'"'s body being different locations on the user'"'"'s body, the scale vector defining a coordinate system being shiftable based on a position of the second reference point and a position of the third reference point when at least one of the second location and the third location on the player'"'"'s body shifts;

define a range of scale values in the coordinate system based on the scale vector;

determine a scale value associated with a position of the first reference point along a gesture axis, the gesture axis associated with the data received from the camera and corresponding to the display axis displayed on the display, the determined scale value being based on the range of scale values in the coordinate system; and

highlight on the display at least one of the plurality of menu items based on the determined scale value.

2. The non-transitory computer readable medium of claim 1 wherein the first reference point corresponds to a hand of the player.

3. The non-transitory computer readable medium of claim 1 wherein the gesture axis is substantially vertical.

4. The non-transitory computer readable medium of claim 1 wherein the gesture axis is substantially parallel to the display axis.

5. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to lock the at least one highlighted menu item in response to the first reference point moving a distance in a direction perpendicular to the gesture axis.

6. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to ignore at least one of horizontal and vertical movement of the first reference point as a function of a deviation window.

7. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to determine a measurement of the first reference point relative to the second reference point.

8. The non-transitory computer readable medium of claim 7 wherein the second reference point is a hip of the player.

9. The non-transitory computer readable medium of claim 1 wherein the second reference point is associated with a hip of the player, and the third reference point is associated with a shoulder of the player.

10. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to:

determine whether the first reference point is located in a scroll zone;

when the first reference point is in the scroll zone, cause the plurality of menu items to scroll.

11. The non-transitory computer readable medium of claim 10 further storing instructions that, when executed by the computer cause the computer to indicate the scroll zone on the display.

12. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to:

define the range of scale values by projecting the scale vector onto a vertical axis.

13. The non-transitory computer readable medium of claim 1 further storing instructions that, when executed by the computer, cause the computer to:

define the range of scale values by projecting the scale vector onto the gesture axis.

14. The non-transitory computer readable medium of claim 1, wherein the determined scale value is outside the range of scale values.

15. A computerized method for execution on a game platform, the method comprising:

displaying on a display a plurality of menu items displaced axially along a display axis;

receiving data from a camera system, the data including a representation of a body of a user;

determining a first reference point associated with a first location on the player'"'"'s body based on the data received from the camera system;

determining a scale vector between a second reference point associated with a second location on the player'"'"'s body and a third reference point associated with a third location on the player'"'"'s body, the first, second, and third locations on the player'"'"'s body being different locations on the user'"'"'s body, the scale vector defining a coordinate system being shiftable based on a position of the second reference point and a position of the third reference point when at least one of the second location and the third location on the player'"'"'s body shifts;

defining a range of scale values in the coordinate system based on the scale vector;

determining a scale value associated with a position of the first reference point along a gesture axis that corresponds to the display axis, the gesture axis associated with the data received from the camera and corresponding to the display axis displayed on the display, the determined scale value being based on the range of scale values in the coordinate system; and

highlighting on the display at least one of the plurality of menu items based on the determined scale value.

16. The method of claim 15 wherein the first reference point corresponds to a hand of the player.

17. The method of claim 15 wherein the gesture axis is substantially vertical.

18. The method of claim 15 wherein the gesture axis is substantially parallel to the display axis.

19. The method of claim 15 further comprising locking the at least one highlighted menu item in response to the first reference point moving a distance in a direction perpendicular to the gesture axis.

20. The method of claim 15 further comprising ignoring at least one of horizontal and vertical movement of the first reference point as a function of a deviation window.

21. The method of claim 15 further comprising determining a measurement of the first reference point relative to the second reference point.

22. The method of claim 21 wherein the second reference point is a hip of the player.

23. The method of claim 15 wherein the second reference point is associated with a hip of the player, and the third reference point is associated with a shoulder of the player.

24. The method of claim 15 further comprising:

determining whether the first reference point is located in a scroll zone;

when the first reference point is in the scroll zone, causing the plurality of menu items to scroll.

25. The method of claim 24 further comprising indicating the scroll zone on the display.

26. The method of claim 15 further comprising:

defining the range of scale values by projecting the scale vector onto a vertical axis.

27. The method of claim 15 further comprising:

defining the range of scale values by projecting the scale vector onto the gesture axis.

28. The method of claim 15 wherein the determined scale value is outside the range of scale values.