GESTURE-BASED USER INTERFACE

• US 20110185309A1
• Filed: 10/27/2010
• Published: 07/28/2011
• Est. Priority Date: 10/27/2009
• Status: Active Grant

• Current Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Original Assignee: Harmonix Music Systems Incorporated (Epic Games, Inc.)
• Inventors: Flury, Marc A., Matheson, Dare, Goldstein, Seth, Challinor, Ryan, Boch, Matthew C., Crooker, Kasson D., Tate, Dean N., Egozy, Eran B.
• Granted Patent: US 10,357,714 B2
• Time in Patent Office: Days
• US Class Current: 715/784
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/814 Musical performances, e.g. by evaluating the player's ability to follow a notation
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

The invention includes methods, systems, computer program products and means for providing an intuitive user interface for interacting with a game console that obviates the need for, or use of, a typical game controller. In one embodiment, the user interface comprises an analog-like mechanism for manipulating a menu and icon system. The menu system allows the user to highlight a menu option or icon using movement along one axis and to activate the menu option or icon via movement on a different axis. In one embodiment, highlighting is associated with a vertical motion and activation is associated with a horizontal motion. The vertical motion can be tied to the vertical movement of a particular appendage, e.g., the up and down movement of the user'"'"'s right arm, and the horizontal motion can be tied to the horizontal movement of that or another appendage, e.g., movement of the right arm or hand across the body from right to left.

Claims

1. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

displaying a menu on the display comprising menu items;

receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;

generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;

projecting the scale vector onto an axis;

assigning a range of values to the projected scale vector based on the two or more joints within the set;

assigning a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicating on the display that the menu item is highlighted.

2. The method of claim 1, further comprising also indicating on the display, the proximity of the value assigned to the joint to the range of values assigned to each menu item

3. The method of claim 2, wherein the proximity is indicated using graphical indications similar to those used for highlighting, in proportion to the proximity.

4. The method of claim 1, wherein the range of values assigned to each menu item is a single, different value for each menu item.

5. The method of claim 1, wherein the axis corresponds to an orientation of a menu displayed on the display.

6. The method of claim 1 further comprising:

receiving one or more updated versions of the 3D skeleton;

determining if positional information of the one or more updated versions of the 3D skeleton corresponds to the beginning of an activation action; and

if the one or more updated versions of the 3D skeleton corresponds to the beginning of an activation action, indicating the beginning of the activation action by altering the highlighted menu item.

7. The method of claim 6, wherein the activation action comprises movement of a joint in a direction substantially perpendicular to the axis.

8. The method of claim 7, wherein the activation action comprises movement of a joint associated with the activation action at least a first pre-determined distance in a direction substantially perpendicular to the axis and less than a second pre-determined distance in a direction substantially parallel to the axis.

9. The method of claim 8, wherein completing the activation action activates the highlighted menu item.

10. The method of claim 7, wherein the activation action comprises movement of a joint associated with the activation action of at least a first distance based on two or more joints in the set of two or more joints of the 3D skeleton, in a direction substantially perpendicular to the axis and less than a second distance in a direction substantially parallel to the axis.

11. The method of claim 10, wherein completing the activation action activates the highlighted menu item.

12. The method of claim 10, wherein the joint associated with the activation action is the same joint as the joint associated with the highlighting action.

13. The method of claim 6, wherein, once the activation action has begun, changes in the value of the joint associated with the highlighting action do not change the highlighted menu item.

14. The method of claim 8, wherein the joint associated with the activation action is the same joint as the joint associated with the highlighting action.

15. The method of claim 6, wherein the highlighted menu item is altered in proportion to completion of the activation action.

16. The method of claim 15, wherein the highlighted menu item is altered by stretching.

17. The method of claim 6, wherein an audible cue is played with characteristics in proportion to completion of the activation action.

18. The method of claim 6, wherein the beginning of the activation action ceases to be indicated if the one or more updated versions of the 3D skeleton ceases to correspond to an activation action.

19. The method of claim 1, wherein the two or more joints that the scale vector is based on comprise a shoulder joint and a hip joint.

20. The method of claim 1, wherein the two or more joints that the scale vector is based on comprise a shoulder joint and a simulated pelvis joint.

21. The method of claim 1, further comprising:

assigning a range of values to a scroll area, the range of values assigned to the scroll area being with respect to the range of values assigned to the projected scale vector;

comparing the value of the joint associated with the highlighting action to the range of values assigned to the scroll area; and

if the value of the joint is within the range of values assigned to the scroll area, scrolling a different set of menu items onto the display.

22. The method of claim 21, wherein the rate at which the different set of menu items is scrolled onto the display is based on the value of the joint in relation to the range of values assigned to the scroll area.

23. The method of claim 21, further comprising assigning a first sub-range of values in the range of values assigned to the scroll area with a slow scrolling operation and assigning a second sub-range of values in the range of values assigned to the scroll area with a fast scrolling operation.

24. The method of claim 23, further comprising

determining if the value of the joint associated with the highlight action is within the sub-range assigned to the slow scrolling operation, and if the value of the joint is within the sub-range assigned to the slow scrolling operation, scrolling the different set of menu items onto the display at a first rate;

and

determining if the value of the joint associated with the highlight action is within the sub-range assigned to the fast scrolling operation, and if the value of the joint is within the sub-range assigned to the fast scrolling operation, scrolling the different set of menu items onto the display at a second rate, faster than the first rate.

25. The method of claim 1, wherein one of the joints of the 3D skeleton is an approximation of one or more joints of the player.

26. The method of claim 1, further comprising:

receiving an updated version of the 3D skeleton;

assigning a second value to the joint associated with the highlighting action, the value being with respect to the range of values assigned to the projected scale vector; and

if the second value is within a range of values assigned to the highlighted menu item or is within a pre-determined range adjacent to the range of values assigned to the highlighted menu item, continuing to highlight the highlighted menu item.

27. The method of claim 26, further comprising, if the second value is not within the range of values assigned to the highlighted menu item and is not within the pre-determined range adjacent to the range of values assigned to the highlighted menu item, no longer highlighting the highlighted menu item.

29. The method of claim 1, further comprising:

receiving an updated version of the 3D skeleton, the updated skeleton'"'"'s set of two or more joints associated with a second spatial position of the two or more reference points on the player'"'"'s body;

updating the scale vector based on the updated skeleton; and

re-projecting the updated scale vector onto the axis.

28. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a camera, the method comprising:

displaying a menu on the display comprising menu items;

receiving a 3D skeleton representing a body of a player, the 3D skeleton comprising joints associated with at least one of the player'"'"'s shoulders, a lower torso area of the player, and at least one of the player'"'"'s wrist;

generating a scale vector based on the distance between the 3D skeleton'"'"'s shoulder joint and an approximation of the player'"'"'s pelvis;

projecting the scale vector onto an axis;

assigning a range of values to the projected scale vector based on the joint associated with the player'"'"'s shoulder and on the approximation of the player'"'"'s pelvis;

assigning a value to a wrist joint, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

comparing the value of the wrist joint to the range of values of each menu item to determine a highlighted menu item; and

indicating on the display that the menu item is highlighted.

30. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

associating a first direction of potential movement with a highlighting action;

associating a second direction of potential movement, different than the first direction, with an activating action, wherein the first and second directions are greater than or equal to a first threshold difference in degrees with respect to a common axis and are less than or equal to a second threshold difference in degrees with respect to the common axis;

determining a first spatial position of a joint within the set of one or more joints;

receiving one or more updated versions of the 3D skeleton;

determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton;

determining, based on the first and second spatial positions of the joint, if the joint has moved in the direction associated with the highlighting action or in the direction associated with the activation action; and

performing the action indicated by the movement of the joint.

31. The method of claim 30 wherein the first threshold difference and the second threshold difference are both ninety degrees.

32. The method of claim 30 wherein the first threshold difference is 75 degrees and the second threshold difference is 105 degrees.

33. The method of claim 30 wherein if the joint has moved in the direction associated with the activation action, preventing a change associated with the highlighting action.

34. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

displaying a menu on the display comprising menu items;

receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

assigning a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position;

assigning a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position;

comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicating on the display that the menu item is highlighted.

35. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

determining a first spatial position of a joint in the set of one or more joints;

assigning an initial positional value to the beginning of an activation action based on the first spatial position of the joint;

assigning a positional value to the completion of the activation action;

receiving one or more updated versions of the 3D skeleton;

determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and

determining if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.

36. A method, executed on a game platform, for interacting with a game, displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:

displaying a scrollable window on the display, the scrollable window comprising a first set of visual data, an area not associated with scrolling a second set of visual data into the scrollable window, and an area associated with scrolling the second set of visual data into the scrollable window;

receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;

generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;

projecting the scale vector onto an axis;

assigning a range of values to the projected scale vector based on the two or more joints within the set;

assigning a value to a joint within the set of two or more joints, where the joint is associated with a scrolling action, the value being with respect to the range of values assigned to the projected scale vector;

assigning a range of values to the area associated with scrolling the visual data, the range of values assigned to the area associated with scrolling being with respect to the range of values assigned to the projected scale vector;

comparing the value of the joint associated with the scrolling action to the range of values assigned to the area associated with scrolling the second set of visual data into the scrollable window; and

if the value of the joint is within the range assigned to the area associated with scrolling the second visual data into the scrollable window, scrolling the second set of visual data into the scrollable window.

37. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:

display, on a display in communication with the data processing apparatus, a menu on the display comprising menu items;

receive a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;

generate a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;

project the scale vector onto an axis;

assign a range of values to the projected scale vector based on the two or more joints within the set;

assign a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

assign a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicate on the display that the menu item is highlighted.

38. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:

display, on a display in communication with the data processing apparatus, a menu on the display comprising menu items;

receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

assign a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position;

assign a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position;

compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicate on the display that the menu item is highlighted.

39. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:

receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

determine a first spatial position of a joint in the set of one or more joints;

assign an initial positional value to the beginning of an activation action based on the first spatial position of the joint;

assign a positional value to the completion of the activation action;

receive one or more updated versions of the 3D skeleton;

determine a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and

determine if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.

40. A system for interacting with a menu system based on input received via a sensor system, the system comprising:

the sensor system for sensing input from a player, the sensor in communication with a game platform;

a display in communication with the game platform;

the game platform, configured to;

display, on the display, a menu on the display comprising menu items;

receive a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;

generate a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;

project the scale vector onto an axis;

assign a range of values to the projected scale vector based on the two or more joints within the set;

assign a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

assign a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicate on the display that the menu item is highlighted.

41. A system for interacting with a menu system based on input received via a sensor system, the system comprising:

the sensor system for sensing input from a player, the sensor in communication with a game platform;

a display in communication with the game platform;

the game platform, configured to;

display, on the display, a menu on the display comprising menu items;

receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

assign a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position;

assign a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position;

compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

indicate on the display that the menu item is highlighted.

42. A system for interacting with a menu system based on input received via a sensor system, the system comprising:

the sensor system for sensing input from a player, the sensor in communication with a game platform;

a display in communication with the game platform;

the game platform, configured to;

receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

determine a first spatial position of a joint in the set of one or more joints;

assign an initial positional value to the beginning of an activation action based on the first spatial position of the joint;

assign a positional value to the completion of the activation action;

receive one or more updated versions of the 3D skeleton;

determine a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and

determine if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.

43. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:

means for displaying a menu on the display comprising menu items;

means for receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;

means for generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;

means for projecting the scale vector onto an axis;

means for assigning a range of values to the projected scale vector based on the two or more joints within the set;

means for assigning a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

means for assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

means for comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

means for indicating on the display that the menu item is highlighted.

44. An apparatus for interacting with a menu system displayed on a display, based on input received via a camera, the apparatus comprising:

means for displaying a menu on the display comprising menu items;

means for receiving a 3D skeleton representing a body of a player, the 3D skeleton comprising joints associated with at least one of the player'"'"'s shoulders, a lower torso area of the player, and at least one of the player'"'"'s wrist;

means for generating a scale vector based on the distance between the 3D skeleton'"'"'s shoulder joint and an approximation of the player'"'"'s pelvis;

means for projecting the scale vector onto an axis;

means for assigning a range of values to the projected scale vector based on the joint associated with the player'"'"'s shoulder and on the approximation of the player'"'"'s pelvis;

means for assigning a value to a wrist joint, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;

means for assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;

means for comparing the value of the wrist joint to the range of values of each menu item to determine a highlighted menu item; and

means for indicating on the display that the menu item is highlighted.

45. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:

means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

means for associating a first direction of potential movement with a highlighting action;

means for associating a second direction of potential movement, different than the first direction, with an activating action, wherein the first and second directions are greater than or equal to a first threshold difference in degrees with respect to a common axis and are less than or equal to a second threshold difference in degrees with respect to the common axis;

means for determining a first spatial position of a joint within the set of one or more joints;

means for receiving one or more updated versions of the 3D skeleton;

means for determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton;

means for determining, based on the first and second spatial positions of the joint, if the joint has moved in the direction associated with the highlighting action or in the direction associated with the activation action; and

means for performing the action indicated by the movement of the joint.

46. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:

means for displaying a menu on the display comprising menu items;

means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

means for assigning a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position;

means for assigning a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position;

means for comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and

means for indicating on the display that the menu item is highlighted.

47. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:

means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body;

means for determining a first spatial position of a joint in the set of one or more joints;

means for assigning an initial positional value to the beginning of an activation action based on the first spatial position of the joint;

means for assigning a positional value to the completion of the activation action;

means for receiving one or more updated versions of the 3D skeleton;

means for determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and

determining if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.