Example-based creation and tuning of motion recognizers for motion-controlled applications

• US 8,051,024 B1
• Filed: 02/25/2010
• Issued: 11/01/2011
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: AiLive, Inc.
• Original Assignee: AiLive, Inc.
• Inventors: Bererton, Curt, Dobson, Daniel, Funge, John, Musick, JR., Charles, Reynolds, Stuart, Tu, Xiaoyuan, Wright, Ian, Yen, Wei
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): RIFKIN, BEN M
• Application Number:
  US12/712,192
• Time in Patent Office: 614 Days
• Field of Search: 706/46
• US Class Current: 706/46
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6009 for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for tuning motion recognizers in motion detections are described. The motion recognizers are created or generated in advance by trained users. The motion recognizers are then loaded into a processing unit that receives motion signals from a handheld motion sensitive device being controlled by an end user. While motion recognition of the motion signals is being made, some or all of the motion recognizers are tuned by the motion signals to make the motion recognizers more adaptive to the user.

Claims

1. A method for using motion recognizers, the method comprising:

loading a set of the motion recognizers that are created in advance by at least one trained user;

receiving motion signals from a hand held motion-sensitive device, where an end user is manipulating the hand held motion-sensitive device in response to a virtual environment being displayed on a display screen;

forming a training set including some or all of the motion signals;

updating one or more of the motion recognizers in the set of motion recognizers with the training set to modify a motion recognition behavior of the one or more motion recognizers; and

using the set of motion recognizers to provide motion recognition signals that one or more of objects in the virtual environment are responsive to.

2. The method as recited in claim 1, wherein the set of the motion recognizers is created by operations including:

causing the trained user to perform motions according to a set of predefined moves;

recording some or all of the motions when the some or all of the motions are qualified in a sense that the some or all of the motions are substantially consistent with desired motions; and

generating each of the motion recognizers by determining corresponding prototypes with best classification rates.

3. The method as recited in claim 2, wherein the motions are made in context of a scene.

4. The method as recited in claim 3, wherein there is a module configured to select the motions and assign corresponding labels thereto, the module provides additional summary information on the motions as to how the hand held motion-sensitive device is supposed to be held.

5. The method as recited in claim 2, wherein the operations further comprise:

calculating a classification distance of each of the motions to prototypes in the motion recognizers;

labeling the each of the motions as undetermined if the classification distance matches none of the prototypes;

or

labeling the each of the motions as a labeled motion associated with one of the prototypes if the classification distance matches only one prototype;

or

labeling the each of the motions as a labeled motion associated with one of some of the prototypes if the classification distance matches some of the prototypes, where the one of some of the prototypes is determined by a smallest classification distance.

6. The method as recited in claim 5, wherein all labeled motions are organized automatically in hierarchy, capacity for each of the motion recognizers and substantially all other information needed to perform classification are created automatically from the motions.

7. The method as recited in claim 1, wherein said forming of the training set including some or all of the motion signals comprises:

receiving the motion signals wirelessly from the hand held motion-sensitive device;

preprocessing the motion signals by a filtering means; and

segmenting the motion signals adaptively according to corresponding magnitudes of the underlying motion signals.

8. The method as recited in claim 1, wherein said loading of the set of the motion recognizers comprises:

receiving the motion recognizers in a portable storage medium;

downloading the motion recognizers via a network;

or

receiving the motion recognizers embedded in a video game.

9. The method as recited in claim 8, wherein said updating one or more of the motion recognizers in the set of motion recognizers with the training set comprises:

calculating classification rates for the motion signals; and

modifying each of the motion recognizers by determining which of the motion signals to add as additional prototypes.

10. The method as recited in claim 8, wherein said updating of one or more of the motion recognizers in the set of motion recognizers with the training set comprises:

calculating a classification distance of each of the motion signals to prototypes stored in the motion recognizers;

labeling the each of the motion signals as unknown or undetermined, if none of the prototypes are within the classification distance;

or

labeling the each of the motion signals with a label associated with an identified prototype, if there is only one prototype is within the classification distance;

or

assigning the each of the motion signals with a label associated with a prototype that is determined by majority vote, if there is more than one of the prototypes within the classification distance; and

adding one or more of the motion signals as prototypes to the one or more of the motion recognizers.

11. The method as recited in claim 1, wherein new motion recognizers are generated within context of a video game played by the end user.

12. The method as recited in claim 11, wherein the training set is augmented by providing more motion signals till motion recognition with the new motion recognizers reaches a level satisfied by the end user.

13. A system for using motion recognizers, the system comprising:

a processing unit;

at least one hand held motion-sensitive device generating motion signals when being manipulated by an end user, the motion signals being transmitted to the processing unit,

wherein the processing unit includes a set of the motion recognizers that are created in advance by at least one trained user and a module configured to perform operations of;

forming a training set including some or all of the motion signals;

tuning one or more of the motion recognizers in the set of motion recognizers based on the training set; and

using the set of motion recognizers to provide motion recognition signals that one or more objects in a virtual environment are responsive to.

14. The system as recited in claim 13, wherein the set of the motion recognizers is created by operations including:

causing the trained user to perform motions according to a set of predefined moves;

recording some or all of the motions when the some or all of the motions are qualified in a sense that the some or all of the motions are substantially consistent with desired motions; and

generating each of the motion recognizers by determining corresponding prototypes with best classification rates.

15. The system as recited in claim 14, wherein the motions are made in context of a scene.

16. The system as recited in claim 15, wherein the module is further configured to select the motions and assign corresponding labels thereto, the module provides additional summary information on the motions as to how the hand held motion-sensitive device is supposed to be held.

17. The system as recited in claim 14, wherein the operations further comprise:

calculating a classification distance of each of the motions to prototypes in the motion recognizers;

labeling the each of the motions as undetermined if the classification distance matches none of the prototypes;

labeling the each of the motions as a labeled motion associated with one of the prototypes if the classification distance matches only one prototype;

or

labeling the each of the motions as a labeled motion associated with one of some of the prototypes if the classification distance matches some of the prototypes, where the one of some of the prototypes is determined by a smallest classification distance; and

adding one or more of the motion signals as prototypes to the one or more of the motion recognizers.

18. The system as recited in claim 17, wherein all labeled motions are organized automatically in hierarchy, capacity for the new motion recognizers and substantially all other information needed to perform classification are created automatically from the training set.

19. The system as recited in claim 13, wherein the training set is formed by operations comprising:

receiving the motion signals wirelessly from the hand-held motion-sensitive device;

preprocessing the motion signals by a filtering means; and

segmenting the motion signals adaptively according to corresponding magnitudes of the underlying motion signals.

20. The system as recited in claim 13, wherein the set of the motion recognizers is received in a portable storage medium, downloaded from the Internet;

or embedded in a video game.

21. The system as recited in claim 20, wherein said tuning one or more of the motion recognizers in the set of motion recognizers based on the training set comprises:

calculating classification rates for the motion signals; and

generating each of the new motion recognizers by determining corresponding prototypes with best vales of the classification rates.

22. The system as recited in claim 20, wherein tuning one or more of the motion recognizers in the set of motion recognizers based on the training set comprises:

calculating a classification distance of each of the motion signals to prototypes stored in the motion recognizers;

labeling the each of the motion signals as unknown or undetermined, if none of the prototypes are within the classification distance;

or

labeling the each of the motion signals with a label associated with an identified prototype, if there is only one prototype is within the classification distance;

or

assigning the each of the motion signals with a label associated with a prototype that is determined by majority vote, if there is more than one of the prototypes within the classification distance.

23. The system as recited in claim 13, wherein new motion recognizers are generated within context of a video game played by the end user.

24. The system as recited in claim 23, wherein the training set is augmented by providing more motion signals till motion recognition with the new motion recognizers reaches a level satisfied by the end user.

25. A system for using motion recognizers, the system comprising:

a processing unit;

at least one hand-held motion sensing device sending motion signals to the processing unit in order to control one or more objects in a virtual environment by an end user;

at least one motion recognizer residing in the processing unit, wherein means to influence which moves are recognizable, which different ways there are to execute a given move, and how tolerant such recognition should be, is to add or subtract examples of motions to or from a training set used to create or update the motion recognizer, wherein the training set is created in advance by a trained user while performing motions according to a set of predefined moves, the processing unit receives the motion signals from the hand-held motion sensing device, and is configured to be responsive to the motion recognizer and compute a motion label for some or all of the motion signals, wherein the motion recognizer is further tunable by the some or all of the motion signals.

26. The system as recited in claim 25, wherein the motion recognizer is to be tuned by an untrained user, and the motion signals are sent from the hand-held motion sensing device as a result of the hand-held motion sensing device being manipulated by the untrained user.

27. The system as recited in claim 26, wherein the processing unit includes a module configured to calculate a classification distance of each of the motion signals to prototypes in the motion recognizer.

28. The system as recited in claim 27, wherein the module is further configured to perform operations of:

labeling the each of the motion signals as undetermined if the classification distance fails to match any of the prototypes;

or

labeling the each of the motion signals as a labeled motion associated with one of the prototypes if the classification distance matches only one prototype;

or

labeling the each of the motion signals as a labeled motion associated with one of some of the prototypes, if there is more than one of the prototypes within the classification distance.

29. The system as recited in claim 25, wherein the motion recognizer is tuned with the motion signals.

30. The system as recited in claim 29, wherein the motion recognizer is repeatedly tuned when recognition rates decline.

31. The system as recited in claim 26, wherein the processing unit is configured to label a motion signal upon receiving only a leading portion of the motion signal to minimize a latency in recognizing a motion pertaining to the motion signal.

32. The system as recited in claim 31, wherein the motion signal is labeled according to a majority vote among all possible prototypes.

33. A method for using motion recognizers, the method comprising:

loading a set of the motion recognizers that are created in advance by at least one trained user;

receiving a beginning portion of a motion signal from a hand held motion-sensitive device being manipulated by an end user in response to a virtual environment being displayed on a display screen; and

preprocessing the beginning portion of the motion signal; and

labeling with reference to the motion recognizer the motion signal without receiving the motion signal in its entirety to minimize latency in motion recognition of the motion signal, wherein the motion recognizer is tunable by the end user using the motion signal.

34. The method as recited in claim 33, further comprising:

forming a training set including the motion signal;

creating at least a new motion recognizer based on the training set; and

updating the set of the motion recognizers to include the new motion recognizer.

35. The method as recited in claim 33, further comprising:

calculating a classification distance of the motion signals to prototypes in the motion recognizers.

36. The method as recited in claim 35, wherein said labeling of the motion signal without receiving the motion signal in its entirety comprises operations of:

labeling the motion signal as undetermined if the classification distance fails to match any of the prototypes;

labeling the motion signal as a labeled motion associated with one of the prototypes if the classification distance matches only one prototype;

or

labeling the motion signal as a labeled motion associated with one of some of the prototypes.

37. A system for using motion recognizers, the system comprising a processing unit loaded with at least one motion recognizer including a set of prototypes to which motion recognition is responsive;

at least one hand-held motion sensing controller sending raw motion signals to the processing unit by an end user in response to a virtual environment being displayed on a display screen;

an adaptive smoother, residing on the processing unit, configured to process the raw motion signals to create processed motion signals that are more robust to unimportant individual variations in motion performance;

and wherein the processing unit is configured to compute a quality measure responsive to a timewarp distance from each of the processed motion signals to prototypes in the motion recognizer to produce a label for the each of the processed motion signals, wherein the motion recognizer is created in advanced by a trained user, and wherein the motion recognizer is tuned with one or more of the raw motion signals.

38. The system as recited in claim 37, wherein the motion signals are sent from the hand-held motion sensing device as a result of the hand-held motion sensing device being manipulated by an untrained user.

39. system as recited in claim 38, wherein the motion recognizer is continuously tuned with subsequent raw motion signals till motion recognition with the tuned motion recognizers reaches a level satisfied by the untrained user.

40. The system as recited in claim 38, wherein the motion recognizer is repeatedly tuned when recognition rates decline.

41. The system as recited in claim 37, wherein the processing unit is configured to label a motion signal upon receiving only a leading portion of the motion signal to minimize a latency in recognizing a motion pertaining to the motion signal.

42. The system as recited in claim 41, wherein the motion signal is labeled according to a majority vote among all possible prototypes.

43. A system for using motion recognizers, the system comprising:

at least one hand-held motion sensing device generating motion signals when the hand-held motion sensing device is being manipulated by an end user in response to a virtual environment being displayed on a display screen; and

a processing unit, loaded with a set of motion recognizers and receiving the motion signals, configured to compute a motion label for each of the motion signals, wherein the processing unit is configured to modify at least one of the motion recognizers so as to update a current motion control scheme and

wherein the motion recognizers are created in advance by at least one trained user, the processing unit is further configured to include a tuning process allowing an untrained user to modify classification performance for one or more classes in each or all of the motion recognizers.

44. The system as recited in claim 43, wherein the motion signals are sent from the hand-held motion sensing device as a result of the hand-held motion sensing device being manipulated by an untrained user.

45. The system as recited in claim 44, wherein the processing unit is further configured to label a motion signal upon receiving only a leading portion of the motion signal to minimize a latency in recognizing a motion pertaining to the motion signal.

46. The system as recited in claim 45, wherein one or more of the motion recognizers are repeatedly tuned when recognition rates decline.

47. The system as recited in claim 44, wherein the processing unit includes a communication means to receive additional motion recognizers.

48. The system as recited in claim 47, wherein the additional motion recognizers are received in a portable storage medium, downloaded via a network;

or embedded in a video game to be executed in the processing unit.