GESTURE-BASED USER INTERFACE
First Claim
1. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
- displaying a menu on the display comprising menu items;
receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body;
generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints;
projecting the scale vector onto an axis;
assigning a range of values to the projected scale vector based on the two or more joints within the set;
assigning a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector;
assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector;
comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and
indicating on the display that the menu item is highlighted.
3 Assignments
0 Petitions
Accused Products
Abstract
The invention includes methods, systems, computer program products and means for providing an intuitive user interface for interacting with a game console that obviates the need for, or use of, a typical game controller. In one embodiment, the user interface comprises an analog-like mechanism for manipulating a menu and icon system. The menu system allows the user to highlight a menu option or icon using movement along one axis and to activate the menu option or icon via movement on a different axis. In one embodiment, highlighting is associated with a vertical motion and activation is associated with a horizontal motion. The vertical motion can be tied to the vertical movement of a particular appendage, e.g., the up and down movement of the user'"'"'s right arm, and the horizontal motion can be tied to the horizontal movement of that or another appendage, e.g., movement of the right arm or hand across the body from right to left.
259 Citations
47 Claims
-
1. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
-
displaying a menu on the display comprising menu items; receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body; generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints; projecting the scale vector onto an axis; assigning a range of values to the projected scale vector based on the two or more joints within the set; assigning a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicating on the display that the menu item is highlighted. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 29)
-
-
28. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a camera, the method comprising:
-
displaying a menu on the display comprising menu items; receiving a 3D skeleton representing a body of a player, the 3D skeleton comprising joints associated with at least one of the player'"'"'s shoulders, a lower torso area of the player, and at least one of the player'"'"'s wrist; generating a scale vector based on the distance between the 3D skeleton'"'"'s shoulder joint and an approximation of the player'"'"'s pelvis; projecting the scale vector onto an axis; assigning a range of values to the projected scale vector based on the joint associated with the player'"'"'s shoulder and on the approximation of the player'"'"'s pelvis; assigning a value to a wrist joint, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; comparing the value of the wrist joint to the range of values of each menu item to determine a highlighted menu item; and indicating on the display that the menu item is highlighted.
-
-
30. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
-
receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; associating a first direction of potential movement with a highlighting action; associating a second direction of potential movement, different than the first direction, with an activating action, wherein the first and second directions are greater than or equal to a first threshold difference in degrees with respect to a common axis and are less than or equal to a second threshold difference in degrees with respect to the common axis; determining a first spatial position of a joint within the set of one or more joints; receiving one or more updated versions of the 3D skeleton; determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; determining, based on the first and second spatial positions of the joint, if the joint has moved in the direction associated with the highlighting action or in the direction associated with the activation action; and performing the action indicated by the movement of the joint. - View Dependent Claims (31, 32, 33)
-
-
34. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
-
displaying a menu on the display comprising menu items; receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; assigning a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position; assigning a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position; comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicating on the display that the menu item is highlighted.
-
-
35. A method, executed on a game platform, for interacting with a menu system displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
-
receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; determining a first spatial position of a joint in the set of one or more joints; assigning an initial positional value to the beginning of an activation action based on the first spatial position of the joint; assigning a positional value to the completion of the activation action; receiving one or more updated versions of the 3D skeleton; determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and determining if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.
-
-
36. A method, executed on a game platform, for interacting with a game, displayed on a display in communication with the game platform, based on input received via a sensor, the method comprising:
-
displaying a scrollable window on the display, the scrollable window comprising a first set of visual data, an area not associated with scrolling a second set of visual data into the scrollable window, and an area associated with scrolling the second set of visual data into the scrollable window; receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body; generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints; projecting the scale vector onto an axis; assigning a range of values to the projected scale vector based on the two or more joints within the set; assigning a value to a joint within the set of two or more joints, where the joint is associated with a scrolling action, the value being with respect to the range of values assigned to the projected scale vector; assigning a range of values to the area associated with scrolling the visual data, the range of values assigned to the area associated with scrolling being with respect to the range of values assigned to the projected scale vector; comparing the value of the joint associated with the scrolling action to the range of values assigned to the area associated with scrolling the second set of visual data into the scrollable window; and if the value of the joint is within the range assigned to the area associated with scrolling the second visual data into the scrollable window, scrolling the second set of visual data into the scrollable window.
-
-
37. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:
-
display, on a display in communication with the data processing apparatus, a menu on the display comprising menu items; receive a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body; generate a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints; project the scale vector onto an axis; assign a range of values to the projected scale vector based on the two or more joints within the set; assign a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; assign a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicate on the display that the menu item is highlighted.
-
-
38. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:
-
display, on a display in communication with the data processing apparatus, a menu on the display comprising menu items; receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; assign a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position; assign a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position; compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicate on the display that the menu item is highlighted.
-
-
39. A computer program product, tangibly embodied in a non-transitory computer readable storage medium, for interacting with a menu system based on input received via a sensor, the computer program product including instructions being operable to cause a data processing apparatus to:
-
receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; determine a first spatial position of a joint in the set of one or more joints; assign an initial positional value to the beginning of an activation action based on the first spatial position of the joint; assign a positional value to the completion of the activation action; receive one or more updated versions of the 3D skeleton; determine a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and determine if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.
-
-
40. A system for interacting with a menu system based on input received via a sensor system, the system comprising:
-
the sensor system for sensing input from a player, the sensor in communication with a game platform; a display in communication with the game platform; the game platform, configured to; display, on the display, a menu on the display comprising menu items; receive a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body; generate a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints; project the scale vector onto an axis; assign a range of values to the projected scale vector based on the two or more joints within the set; assign a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; assign a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicate on the display that the menu item is highlighted.
-
-
41. A system for interacting with a menu system based on input received via a sensor system, the system comprising:
-
the sensor system for sensing input from a player, the sensor in communication with a game platform; a display in communication with the game platform; the game platform, configured to; display, on the display, a menu on the display comprising menu items; receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; assign a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position; assign a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position; compare the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and indicate on the display that the menu item is highlighted.
-
-
42. A system for interacting with a menu system based on input received via a sensor system, the system comprising:
-
the sensor system for sensing input from a player, the sensor in communication with a game platform; a display in communication with the game platform; the game platform, configured to; receive a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; determine a first spatial position of a joint in the set of one or more joints; assign an initial positional value to the beginning of an activation action based on the first spatial position of the joint; assign a positional value to the completion of the activation action; receive one or more updated versions of the 3D skeleton; determine a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and determine if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.
-
-
43. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:
-
means for displaying a menu on the display comprising menu items; means for receiving a 3D skeleton having a set of two or more joints associated with a first spatial position of two or more reference points on a player'"'"'s body; means for generating a scale vector based on two or more of the joints of the 3D skeleton within the set of two or more joints; means for projecting the scale vector onto an axis; means for assigning a range of values to the projected scale vector based on the two or more joints within the set; means for assigning a value to a joint within the set of two or more joints, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; means for assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; means for comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and means for indicating on the display that the menu item is highlighted.
-
-
44. An apparatus for interacting with a menu system displayed on a display, based on input received via a camera, the apparatus comprising:
-
means for displaying a menu on the display comprising menu items; means for receiving a 3D skeleton representing a body of a player, the 3D skeleton comprising joints associated with at least one of the player'"'"'s shoulders, a lower torso area of the player, and at least one of the player'"'"'s wrist; means for generating a scale vector based on the distance between the 3D skeleton'"'"'s shoulder joint and an approximation of the player'"'"'s pelvis; means for projecting the scale vector onto an axis; means for assigning a range of values to the projected scale vector based on the joint associated with the player'"'"'s shoulder and on the approximation of the player'"'"'s pelvis; means for assigning a value to a wrist joint, where the joint is associated with a highlighting action, the value being with respect to the range of values assigned to the projected scale vector; means for assigning a range of values to each menu item, the range of values assigned to each menu item being with respect to the range of values assigned to the projected scale vector; means for comparing the value of the wrist joint to the range of values of each menu item to determine a highlighted menu item; and means for indicating on the display that the menu item is highlighted.
-
-
45. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:
-
means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; means for associating a first direction of potential movement with a highlighting action; means for associating a second direction of potential movement, different than the first direction, with an activating action, wherein the first and second directions are greater than or equal to a first threshold difference in degrees with respect to a common axis and are less than or equal to a second threshold difference in degrees with respect to the common axis; means for determining a first spatial position of a joint within the set of one or more joints; means for receiving one or more updated versions of the 3D skeleton; means for determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; means for determining, based on the first and second spatial positions of the joint, if the joint has moved in the direction associated with the highlighting action or in the direction associated with the activation action; and means for performing the action indicated by the movement of the joint.
-
-
46. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:
-
means for displaying a menu on the display comprising menu items; means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; means for assigning a range of values to each menu item, the range of values assigned to each menu item being based on each menu item'"'"'s respective position; means for assigning a value to a joint within the set of one or more joints, where the joint is associated with a highlighting action, the value being based on the joint'"'"'s position; means for comparing the value of the joint associated with the highlighting action to the range of values of each menu item to determine a highlighted menu item; and means for indicating on the display that the menu item is highlighted.
-
-
47. An apparatus for interacting with a menu system displayed on a display, based on input received via a sensor, the apparatus comprising:
-
means for receiving a 3D skeleton having a set of one or more joints associated with a first spatial position of one or more reference points on a player'"'"'s body; means for determining a first spatial position of a joint in the set of one or more joints; means for assigning an initial positional value to the beginning of an activation action based on the first spatial position of the joint; means for assigning a positional value to the completion of the activation action; means for receiving one or more updated versions of the 3D skeleton; means for determining a second spatial position of the joint within the set of one or more joints based on the one or more updated versions of the 3D skeleton; and determining if the second spatial position of the joint corresponds to the positional value assigned to the completion of the activation action, and if so, executing the activation action.
-
Specification