Generating motion recognizers for arbitrary motions for video games and tuning the motion recognizers to the end user

• US 7,702,608 B1
• Filed: 07/14/2006
• Issued: 04/20/2010
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Bererton, Curt, Funge, John, Reynolds, Stuart, Wright, Ian, Dobson, Daniel, Tu, Xiaoyuan, Musick, Charles, Yen, Wei
• Primary Examiner(s): Vincent, David R
• Assistant Examiner(s): Rifkin, Ben M
• Application Number:
  US11/486,997
• Time in Patent Office: 1,376 Days
• Field of Search: 706/46
• US Class Current: 706/46
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6009 for importing or creating game content, e.g. authoring tools during game development, adapting content to different platforms, use of a scripting language to create content
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Generating motion recognizers from example motions, without substantial programming, without limitation to any fixed set of well-known gestures, and without limitation to motions that occur substantially in a plane, or are substantially predefined in scope. From example motions for each class of motion to be recognized, a system automatically generates motion recognizers using machine learning techniques. Those motion recognizers can be incorporated into an end-user application, with the effect that when a user of the application supplies a motion, those motion recognizers will recognize the motion as an example of one of the known classes of motion. Motion recognizers can be incorporated into an end-user application; tuned to improve recognition rates for subsequent motions to allow end-users to add new example motions.

Claims

1. A system for recognizing motions, the system comprising:

a computing unit configured to include one or more motion recognizer generators that generate a set of motion recognizers for use in video games in response to a first training set of first motion signals; and

at least one first handheld motion-sensitive device generating the first training set of first motion signals when the first handheld motion-sensitive device is being manipulated by a trainer, each of the first motion signals describing a motion of the first handheld motion-sensitive device over a period of time, each of the motion recognizers configured to generate a motion recognition signal in response to a second motion signal from a second handheld motion-sensitive device,

wherein the computing unit is configured to tune the motion recognizers in response to one or more of a second set of motion signals generated from the second handheld motion-sensitive device when the second handheld motion-sensitive device is manipulated by an end user, each of the second set of motion signals describing at least one motion of the second handheld motion-sensitive device over a period of time.

2. The system as recited in claim 1, wherein a time warp distance is calculated from the second motion signal to one or more motion prototypes stored in one or more of the motion recognizers to determine a subset of the prototypes when the time warp distance is below a predefined distance threshold, and wherein motion recognition of the second handheld motion-sensitive device is responsive to the subset of the prototypes.

3. The system as recited in claim 1, wherein the second motion signal is an actual motion signal from the second handheld motion-sensitive device used to interact with a game integrated with some or all of the motion recognizers.

4. The system as recited in claim 1, wherein at least one of the motion recognizers generated in response to the first training set of motion signals includes a generalized version of one or more of the first motion signals is a prototype, to which classification will be responsive.

5. The system as recited in claim 3, wherein the second motion signal or one or more of the second set of motion signals is used to tune one or more motions from the training set into a generalized version of the one or more motions from the training set.

6. The system as recited in claim 1, wherein at least one of the motion recognizers generates a recognition signal in response to a prefix of the second motion signal, when a time warp distance calculated from the prefix of the second motion signal to one or more motion prototypes stored in a classifier is compared to a modified distance threshold, leading to an effect of providing relatively low latency incremental motion recognition.

7. The system as recited in claim 1, 2, 3, or 6, wherein,

one or more of the first motion signals in the training set represent one or more motions performed by one or more people involved in game development,

the second motion signal or one of the second set of motion signals represents one or more motions performed by one or more game players,

one or more of the motion recognizers are coupled to a game and are used to classify the second motion,

one or more resulting classifications are coupled to events in the game.

8. The system as recited in claim 1, wherein the first training set of motion signals are a set of raw motion signals, each describing one or more motions of the first handheld motion-sensitive device.

9. The system as recited in claim 8, wherein the computing unit is configured to include a component to process the raw motion signals to filter out undesired effect therein.

10. The system as recited in claim 9, wherein the processing of the raw motion signals includes compressing:

the raw motion signals, adaptively sampling the raw motion signals, modifying the raw motion signals to make the raw motion signals more similar to one or more previously observed motions, or filtering the raw motion signals in order to make the raw motion signals easier to store or recognize.

11. The system as recited in claim 8, wherein the raw motion signals are generated in, besides the first handheld motion-sensitive device, one or more input devices responsive to one or more humans, animals or machines, an output of executed program code, the raw motion signals generated at different locations, dates and times stored on storage media and made available locally or over a network.

12. The system as recited in claim 8, wherein the raw motion signals are generated in one or more input devices that include one or more accelerometers and transmit the raw motion signals wirelessly to the computing unit configured to execute an end user application.

13. The system as recited in claim 12, wherein one of the input devices is a controller handheld and moved around manually.

14. The system as recited in claim 4, wherein the computing unit is configured to include a motion recognizer tuner that modifies a previously generated motion recognizer in response to an additional training set.

15. The system as recited in claim 14, wherein the motion recognizer tuner tunes one or more of the motion recognizers to make them more responsive to one or more motions included in the additional training set, with an effect of improving recognition performance on the second motion signals generated from a substantially similar source as the additional training set.

16. The system as recited in claim 15, wherein the additional training set includes one or more motions that represent motions of a celebrity;

and the resulting tuned recognizer corresponds to a motion recognizer henceforth associated with the celebrity.

17. The system as recited in claim 15, wherein the motion recognizer tuner tunes one or more of the first motion signals, and provides representative prototypes in place of or in addition to one or more of the prototypes corresponding to one or more of the motion recognizers being tuned.

18. The system as recited in claim 14, wherein the additional training set is responsive to feedback from at least one earlier collected motion, with an effect of tuning the motion recognizers accordingly.

19. The system as recited in claim 14, wherein the motion recognizer tuner is capable of performing at least one of the following:

causing the motion recognizers to be responsive to the additional training set, removing the motion recognizers, and

merging the motion recognizers.

20. The system as recited in claim 1, wherein one or more of the motion recognizers generate motion recognition signals for one or more new motion signals in response to an incremental distance comparison to potential representative prototypes.

21. The system as recited in claim 1 further comprising:

memory or mass storage coupled to the computing unit, wherein the memory or mass storage is accessible by another computing device for use of the generated motion recognizers therein.

22. The system as recited in claim 1, wherein

one or more of the motion recognizer generators are sent over a network for use on another computing device.

23. The system as recited in claim 22, wherein the motion recognition signal includes a class label to identify a particular motion generated from the second handheld motion-sensitive device.

24. The system as recited in claim 23, wherein the motion recognition signal includes a list of class labels ranked in response to a measure of relative likelihood that one of the second set of motion signals is a member of a class.

25. The system as recited in claim 23, wherein the motion recognition signal includes at least one response indicating that none of the motion recognizers recognize the second motion signal or one of the second set of motion signals.

26. The system as recited in claim 1, wherein the motion recognition signal is a sequence of numbers that represent X, Y, Z accelerations of the second handheld motion-sensitive device.

27. The system as recited in claim 2, wherein, as the second handheld motion-sensitive device is caused to move around, the second set of motion signals result in a plurality of motions ranked according to respective classification rates.

28. The system as recited in claim 26, wherein the motion recognizer generators create the motion recognizers in varying capacity.

29. The system as recited in claim 1 further comprising:

a coupled application being executed in another computing unit, wherein one or more end users of the application are able to redefine one or more of the motion recognizers in response to either a new training set or a modified existing training set.

30. The system as recited in claim 29, wherein

the coupled application includes a game played by one or more game players with one or more controllers generating various motion signals,

the second motion signal or one of the second set of motion signals represents one or more motions performed by the one or more game players,

one or more of the motion recognizers are used to classify one or more motions performed by the one or more players, and

one or more resulting classifications are coupled to events in the game.

31. The system as recited in claim 30, wherein the motion recognizers generated in response to the first training set of motion signals represent motions of a first set of players which are used to recognize motions generated by a different set of players, with an effect that the game is responsive to one or more of the different sets of players mimicking one or more of the motions of the first set of players.

32. The system as recited in claim 29, wherein,

one or more of the second set of motion signals represent one or more motions performed by one or more disabled people,

one or more of the motion recognizers are coupled to an application to learn a correspondence between motions a disabled person is able to perform and a meaning associated with those motions, and

one or more of the motion recognizers are used to classify one or more of the motions performed by the disabled users of the application.

33. The system as recited in claim 1, wherein one or more of the generated motion recognizers are coupled to a training program designed to guide, or help, or teach people to learn a physical skill.

34. The system as recited in claim 1, wherein one or more of the generated motion recognizers are coupled to an animal behavior monitoring application.

35. The system as recited in claim 1, wherein one or more of the generated motion recognizers are coupled to a law enforcement monitoring application.

36. The system as recited in claim 1, wherein one or more of the motion recognizers provide feedback in response to a measure of comparison between possibley new motions and a set of motions used to generate or tune the motion recognizers.

37. A method for recognizing motions, the method comprising:

generating a set of motion recognizers for use in videogames in response to a first training set of first motion signals; and

associating each of the motion recognizers with one or more of the first motion signals, wherein each of the first set of motion signals describes a motion of a trainer manipulating at least one first handheld motion-sensitive device over a period of time; and

generating for each of the motion recognizers a motion recognition signal in a computing unit in response to a second motion signal from a second handheld motion-sensitive device;

generating a second set of motion signals from the second handheld motion-sensitive device when the second handheld motion-sensitive device is manipulated by an end user, wherein each of the second set of motion signals describes at least one motion of the second handheld motion-sensitive device over a period of time; and

tuning the motion recognizers in response to one or more of the second set of motion signals.

38. The method as recited in claim 37, wherein a time warp distance is calculated from the second motion signal to one ore more motion prototypes stored in one or more of the motion recognizers to determine a subset of the prototypes when the time warp distance is below a predefined distance threshold, and wherein motion recognition of the second handheld motion-sensitive device is responsive to the subset of the prototypes.

39. The method as recited in claim 37, wherein the computing unit is configured to determine, for at least one of the motion recognizers, a set of motion prototypes that are representative of the first motion signals associated with the at least one of the motion recognizers.

40. The method as recited in claim 37, wherein at least one of the motion recognizers generates a recognition signal in response to a prefix of the second motion signal, when a time warp distance calculated from the second motion signal to one or more motion prototypes stored in a motion recognizer is compared to a modified distance threshold, leading to an effect of providing relatively low latency incremental motion recognition.

41. The method as recited in claim 37, wherein the first training set of first motion signals are a set of raw motion signals, each describing one or more motions of the first handheld motion-sensitive device.

42. The method as recited in claim 41, further comprising:

processing the raw motion signals from motion sensors enclosed in the first handheld motion-sensitive device being manipulated by a trainer.

43. The method as recited in claim 42, wherein said processing of the raw motion signals includes

adaptively sampling the raw motion signals

with an effect of making the raw motion signals easier to be stored or recognized.

44. The method as recited in claim 38 further comprising:

tuning one of the previously generated motion recognizers in response to an additional training set.

45. The method as recited in claim 44, wherein the turning of the previously generated motion recognizer is performed by a motion recognizer tuner that tunes one or more of the motion recognizers to make them more responsive to one or more motions included in the additional training set, resulting in an effect of improving recognition performance on motion signals subsequently generated from a substantially similar source as the additional training set.

46. The method as recited in claim 44, wherein said tuning of the previously generated motion recognizer includes:

selecting one or more of the motion signals from the additional training set as representative prototypes in place of or in addition to one or more of the prototypes in one or more of the motion recognizers being tuned.

47. The method as recited in claim 44, further comprising:

capturing motions responsive to feedback from an entity involved in collecting the motions, with an effect of tuning the motion recognizers to be either more specific or less specific.

48. The method as recited in claim 45, wherein the motion recognizer tuner performs at least one of operations:

adding new motion recognizers responsive to the additional training set;

removing the motion recognizers;

or

merging the motion recognizers.

49. The method as recited in claim 38, wherein one or more motion recognizers perform calculating an incremental measure of distance comparison between one or more new motion signals and potential representative prototypes.

50. The method as recited in claim 37 including:

saving one or more of the motion recognizers in a storage medium; and

making the generated motion recognizers available for use on another computing device.

51. The method as recited in claim 37, including:

sending one or more of the generated motion recognizers over a network for use on one or more other computing devices.

52. The method as recited in claim 51 including:

computing a class label as a recognition signal for the second motion signal.

53. The method as recited in claim 52 including:

computing a list of possible class labels ranked by a measure of relative likelihood that the second motion signal is a member of a class.

54. The method as recited in claim 37, wherein the second motion signal is a sequence of numbers that represent X, Y, Z accelerations of the second handheld motion-sensitive device and further including:

determining if one of the motion recognizers does not satisfactorily recognize the motion signal.

55. The method as recited in claim 54, wherein as the second handheld motion-sensitive device is caused to move around, the second motion signal results in a plurality of motions ranked according to respective classification rates.

56. The method as recited in claim 37, wherein as the second handheld motion-sensitive device is caused to move around, the second motion signal results in a plurality of motions ranked according to respective classification rates.

57. The method as recited in claim 37 including specifying a capacity for each of the generated motion recognizers.

58. The method as recited in claim 37 including coupling an application, wherein one or more end users of the application are able to redefine one or more of the motion recognizers in response to a new training set of motion signals or a modified existing training set of motion signals.

59. The method as recited in claim 58, including:

generating the motion recognizers in response to one or more of the motion signals in the new training set or the modified existing training set created by the end user.

60. The method as recited in claim 58, wherein Wherein the one or more end users are physically challenged.

61. The method as recited in claim 37 including:

generating the motion recognizers in response to one or more of the motion signals in a training set representing one or more motions performed by teachers possessing some physical skills, with the effect that one or more students can attempt to learn that same or similar physical skill by attempting to have their motions correctly classified by those motion recognizers.

62. The method as recited in claim 37 including:

generating the motion recognizers in response to one or more of the motion signals in a training set representing one or more behaviors performed by animals.

63. The method as recited in claim 62 including:

performing behavior modification on at least one of those animals in response to an output of one or more of the motion recognizers.

64. The method as recited in claim 37 including:

generating the motion recognizers representing one or more behaviors performed by people, wherein the behaviors are prescribed or proscribed.

65. The method as recited in claim 64 including:

monitoring one or more persons for these behaviors that are prescribed or proscribed.

66. The method as recited in claim 37, further comprising providing feedback that includes at least one measure of similarity or difference, on how sets of motions compare to motions earlier used to generate or tune the motion recognizers.