Systems and methods for personalized motion control

• US 20110044501A1
• Filed: 02/22/2010
• Published: 02/24/2011
• Est. Priority Date: 07/14/2006
• Status: Abandoned Application

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: KAWANO, Yoichiro, Reynolds, Stuart, Tu, Xiaoyuan, Wright, Ian, Yen, Wei, Wilkinson, Dana, Musick, JR., Charles, Powers, III, William Robert
• Application Number:
  US12/709,520
• Publication Number:
  US 20110044501A1
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

End users, unskilled in the art, generating motion recognizers from example motions, without substantial programming, without limitation to any fixed set of well-known gestures, and without limitation to motions that occur substantially in a plane, or are substantially predefined in scope. From example motions for each class of motion to be recognized, a system automatically generates motion recognizers using machine learning techniques. Those motion recognizers can be incorporated into an end-user application, with the effect that when a user of the application supplies a motion, those motion recognizers will recognize the motion as an example of one of the known classes of motion. Motion recognizers can be incorporated into an end-user application; tuned to improve recognition rates for subsequent motions to allow end-users to add new example motions.

Claims

1. A method for creating and using motion recognizers, the method comprising:

receiving a training set created by an end user without reference to a predefined set of allowed motions, the training set including a first set of motion signals characterizing at least one type of motion executed over some period of time;

constructing at least one of the motion recognizers automatically from the training set, wherein;

(1) substantially all parameters needed to create the motion recognizers that are ad-hoc and perform motion recognition are determined automatically;

(2) means to influence which moves are recognizable is to add new examples of motions to or subtract some of the motion signals from the training set; and

performing motion recognition with the at least one of the motion recognizers by classifying a second set of motion signals

2. The method as recited in claim 1, wherein the constructing of the at least one of the motion recognizers and the performing of the motion recognition occur at the same time on a same device.

3. The method as recited in claim 1 wherein the motion signals are generated from at least one motion sensitive device responsive to human motion over time.

4. The method as recited in claim 1, wherein both the motion recognizers and the motion recognition are responsive to both static poses and dynamic motions of the end user.

5. The method as recited in claim 1, further comprising:

processing the motion signals so that only interesting portions of the motion signals are provided to the constructing of the at least one of the motion recognizers, where the interesting portions of the motion signals include one or more of;

a relative magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a relative magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time; and

a relatively large overall time has passed.

6. The method as recited in claim 1, further comprising:

updating the motion recognizers with one or more motions from the second set of motion signals if the classification distance from such a motion to a prototype in a class of one of the motion recognizers is within a predefined threshold.

7. The method as recited in claim 1, wherein the end user moves a motion sensitive device without a reference to a predefined set of allowed motions, and without a predefined notion of acceptable ways to move the motion sensitive device.

8. The method as recited in claim 7, wherein a motion recognizer maker for constructing the motion recognizers is built into an application to be executed in the motion sensitive device itself or a base unit that communicates with the motion sensitive device.

9. The method as recited in claim 8, wherein each of the motion recognizers is created in accordance with three parameters:

a slack, a capacity, and a start, where the three parameters are automatically set.

10. The method as recited in claim 9, wherein the motion recognizers include prototypes that have been generalized by slack distances, and the recognizer maker is configured to choose most effective prototypes and distances to classify the motion signals in the training set.

11. The method as recited in claim 1, wherein the motion recognizers are created by a motion recognizer maker that computes a small fraction of the pair-wise distances between training set examples that are needed for optimal prototype selection, and selects which examples in the training set become prototypes in the motion recognizer by choosing the examples with the best chance of improving classification rates the most, based on analysis of actual and approximate pair-wise distances between the chosen prototypes and the rest of the training set, the motion recognizers are responsive to the training set and provide optimal or nearly optimal recognition rates given the training data at hand.

12. The method as recited in claim 11, wherein a small fraction of pair-wise distances that get computed are selected by:

computing sufficient pair-wise distances between training examples in each class to form rough intra-class clusters based on proximity;

assigning all remaining motion signals to a closest cluster in its class;

choosing a cluster centroid for each cluster and computing all pair-wise distances between cluster centroids;

using triangle inequality and the already computed pair-wise distances to approximate distances between examples that have not been computed;

computing additional pair-wise distances for edge cases where the triangle inequality breaks down.

13. A system for creating and using motion recognizers, the system comprising:

at least one hand-held motion sensing device producing a first set of motion signals;

a memory space for storing at least one motion recognizer that is ad-hoc, and at least one training set created by an end user without reference to a predefined set of allowed motions, the training set including a second set of motion signals characterizing at least one motion executed over some period of time; and

a first processing unit with a recognizer maker that is configured to automatically build the at least one motion recognizer from the at least one training set; and

a second processing unit that receives the motion signals from the at least one hand-held motion sensing device, and executes a recognition runtime library which, responsive to the at least one motion recognizer, computes a motion label for the motion signals.

14. The system as recited in claim 13, further comprising a processor configured to process the motion signals so that only interesting portions of the motion signals are provided to the motion recognizer maker, where the interesting portions of the motion signals include one or more of:

a relative magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a relative magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time;

or

a relatively large overall time has passed.

15. The system as recited in claim 14, wherein the memory space further stores identifiers, each labeling one of the processed motion signals with an identifier, and motion classes, each of the processed motion signals classified into one of the classes by a false positive rate that indicates a classification distance to prototypes already in one of the classes.

16. The system as recited in claim 15, wherein the class is updated with the each of the processed motion signals if the classification distance is within a predefined threshold.

17. The system as recited in claim 14, wherein the processor is configured to create one of the motion recognizers and perform the motion recognition substantially at the same time.

18. The system as recited in claim 13, wherein the motion signals are generated from at least one motion sensitive device responsive to human motion over time.

19. The system as recited in claim 13, wherein both the motion recognizers and the motion recognition are responsive to both static poses and dynamic motions of the end user.

20. The system as recited in claim 13, further comprising:

processing the motion signals so that only interesting portions of the motion signals are provided to the constructing of the at least one of the motion recognizers, where the interesting portions of the motion signals include one or more of;

a relative magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a relative magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time;

or

a relatively large overall time has passed.

21. The system as recited in claim 13, further comprising:

updating the motion recognizers with one or more motions from the second set of motion signals if the classification distance from such a motion to a prototype in a class of one of the motion recognizers is within a predefined threshold.

22. The system as recited in claim 13, wherein the end user moves a motion sensitive device without a reference to a predefined set of allowed motions, and without a predefined notion of acceptable ways to move the motion sensitive device.

23. The system as recited in claim 22, wherein a motion recognizer maker for constructing the motion recognizers is built into an application to be executed in the motion sensitive device itself or a base unit that communicates with the motion sensitive device.

24. The system as recited in claim 23, wherein each of the motion recognizers is created in accordance with three parameters:

a slack, a capacity, and a start, where the three parameters are automatically set.

25. The system as recited in claim 24, wherein the motion recognizers include prototypes that have been generalized by slack distances, and the recognizer maker is configured to choose most effective prototypes and distances to classify the motion signals in the training set.

26. The system as recited in claim 13, wherein the motion recognizers are created by a motion recognizer maker that

computes a small fraction of the pair-wise distances between training set examples that are needed for optimal prototype selection, and

selects which examples in the training set become prototypes in the motion recognizers by choosing the examples with the best chance of improving classification rates the most, based on analysis of actual and approximate pair-wise distances between the chosen prototypes and the rest of the training set,

wherein the motion recognizers are responsive to the training set and provide optimal or nearly optimal recognition rates per the training data.

27. The system as recited in claim 26, wherein a small fraction of pair-wise distances that get computed are selected by:

computing sufficient pair-wise distances between training examples in each class to form rough intra-class clusters based on proximity;

assigning all remaining motion signals to a closest cluster in its class;

choosing a cluster centroid for each cluster and computing all pair-wise distances between cluster centroids;

using triangle inequality and the already computed pair-wise distances to approximate distances between examples that have not been computed;

computing additional pair-wise distances for edge cases where the triangle inequality breaks down.

28. The system as in claim 13 further comprising a motion control service layer residing on the processing unit that contains a recognition runtime library, and actively manages at least one motion recognizer, and services connections with at least one motion sensitive application on the processing unit, wherein motion signals from the processing unit are independently and simultaneously processed by the motion control service layer in a manner responsive to each related motion recognizer to answer motion control-related queries from each of the motion sensitive applications it is servicing

29. The system as in claim 13, wherein each of the motion signals is an incoming motion signal stream that is automatically segmented, the second processing unit is configured to use a motion start predictor to segment the incoming motion signal stream as part of a motion recognition process in which the second processing unit computes a motion label for the incoming motion signal.

30. A method for creating motion recognizers, the method comprising:

receiving a training set of a first set of motion signals characterizing at least one type of motion executed over some period of time;

constructing at least one motion recognizer automatically from the training set, wherein when used by a recognition runtime library, the motion recognizers support motion recognition on a second set of motion signals); and

computing automatically from the training set at least one of;

(3) a set of slack parameters, which is used to control per-class classification tolerances of the motion recognizer without adding or deleting motion signals from the training set, as a function of (i) overall classification rates, (ii) a difference in per-class classification rates, or (iii) a desired “

undetermined”

classification rate;

(4) a capacity parameter, which is used to control a recognition capacity of the motion recognizer, as a function of (i) number of classes of the motion recognizer, (ii) required classification rates of each class, or (iii) a desired “

undetermined”

classification rate;

(5) a confusion matrix, which is used to guide an interactive use of a recognizer maker by indicating which motion classes in the training set need to be updated with new motion signals or redesigned completely.

31. The method as recited in claim 30, wherein at least one of the slack parameters, and the capacity parameter, is further adjustable by an end user.

32. The method as recited in claim 30, wherein at least one of the slack parameters is computed to equalize per-class classification rates by using an output of a confusion matrix built during the constructing of the motion recognizer to identify which classes are interfering with a proper classification of which other classes, then either adjusting a slack of the over-tolerant class to be less tolerant, or adjusting the slack of the less tolerant class to exhibit a higher classification tolerance.

33. The method as recited in claim 30, wherein during the constructing of the motion recognizer, the capacity parameter is computed by choosing a smallest value for capacity that maximizes an overall predicted classification rate of the motion recognizer while maintaining an undetermined classification rate in an acceptable range.

34. The method as recited in claim 30, wherein the confusion matrix provides a set of class to class false positive and false negative ratios which are analyzed and used to inform the user when and how to change or modify his/her current move set design by indicating which classes are at fault when one or more of the following conditions exist:

(i) two or more classes in the motion recognizer are too close to each other, causing both classes to be hard to recognize;

(ii) one or more classes in the motion recognizer is falsely classifying too many examples, lowering the recognition rates of one of more neighboring classes;

(iii) one or more classes is meeting with poor recognition rates.

35. The method as recited in claim 30, wherein a likelihood for returning an “

undetermined”

label for any given motion signal per the motion recognizer is adjustable, and wherein;

a cost function controls a quality of any proposed classification boundary during the constructing of the motion recognizer;

said cost function is responsive to relative distances between the motion signal and prototypes in the motion recognizer; and

said cost function increases or decreases a cost of distance thereby increasing or decreasing likelihoods of returning an undetermined label for any given input motion signal.

36. A system for creating motion recognizers, the system comprising:

at least one motion sensing device producing a first set of motion signals;

a memory space for storing at least one motion recognizer, and at least one training set including a second set of motion signals characterizing at least one motion executed over some period of time; and

a first processing unit that receives the first set of motion signals from the at least one motion sensing device, and executes a recognition runtime library which, responsive to the at least one motion recognizer, computes a motion label for the first set of motion signals; and

a second processing unit with a recognizer maker configured to automatically build the at least one motion recognizer from the at least one training set and additionally computes automatically from the training set at least one of;

(3) a set of slack parameters, which is used to control per-class classification tolerances of the motion recognizer without adding or deleting any of the second set of motion signals from the training set, as a function of (i) overall classification rates, (ii) a difference in per-class classification rates, or (iii) a desired “

undetermined”

classification rate;

(4) a capacity parameter, which is used to control recognition capacity of the motion recognizer, as a function of (i) number of classes of the motion recognizer, (ii) required classification rates of each class, or (iii) a desired “

undetermined”

classification rate;

(5) a confusion matrix, which is used to guide interactive use of the recognizer maker by indicating which motion classes in the training set need to be updated with new motion signals or redesigned completely.

37. The system as recited in claim 36, wherein a fine grained control of motion recognition is provided, the motion recognition performed on the first processing unit is configured to adjust a likelihood for returning an “

undetermined”

label for any given motion signal with one or more of;

(i) adjusting the likelihood of returning undetermined by modifying a cost of distance when the motion recognizer is being created;

or

(ii) creating an undetermined class and making the undetermined class responsive to one or more examples labeled “

undetermined”

in the training set.

38. A method for creating motion recognizers, the method comprising:

receiving motion signals as a training set of data from one or more motion sensitive devices, each of the motion signals characterizing at least one type of motion executed over some period of time;

recording and retaining an envelope of data for each of the motion signals including data before a start and after an end of the motion characterized in each of the motion signals;

analyzing each of the motion signals to build a motion start classifier that predicts the start of a motion based on features including differences in motion signal activities before, during and after the start of each of the motion signals in the training set; and

labeling an incoming motion signal stream automatically with a motion start when the motion start classifier indicates a motion has started.

39. The method as recited in claim 38, wherein the labeling of the incoming motion signal stream happens either when performing motion recognition or when creating one or more of the motion recognizers.

40. The method as recited in claim 39, further comprising processing the incoming motion signal stream so that only interesting portions of the incoming motion signal stream are used in the motion recognition or for creating one or more of the motion recognizers, where the interesting portions of the incoming motion signal stream include one or more of:

a relative magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a relative magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time;

or

a relatively large overall time has passed.

41. The method as recited in claim 40, further comprising updating the motion recognizers with one or more motions from the second set of motion signals if the classification distance from such a motion to a prototype in a class of one of the motion recognizers is within a predefined threshold.

42. The method as recited in claim 38, wherein the one or more motion sensitive devices are manipulated by one or more end users respectively without a reference to a predefined set of allowed motions, and without a predefined notion of acceptable ways to move the motion sensitive device.

43. The method as recited in claim 38, wherein a motion recognizer maker for constructing the motion recognizers is built into an application to be executed in a motion sensitive device or a base unit that communicates with the motion sensitive device.

44. The method as recited in claim 43, wherein each of the motion recognizers is created in accordance with three parameters:

a slack, a capacity, and a start, where the three parameters are automatically set.

45. The method as recited in claim 44, wherein the motion recognizers include prototypes that have been generalized by slack distances, and the recognizer maker is configured to choose most effective prototypes and distances to classify the motion signals in the training set.

46. The method as recited in claim 38, wherein the motion recognizers are created by a motion recognizer maker that

computes a small fraction of the pair-wise distances between training set examples that are needed for optimal prototype selection, and selects which examples in the training set become prototypes in the motion recognizer by choosing the examples with the best chance of improving classification rates the most, based on analysis of actual and approximate pair-wise distances between the chosen prototypes and the rest of the training set;

the motion recognizers are responsive to the training set and provide optimal or nearly optimal recognition rates given the training data at hand.

47. The method as in claim 46, wherein a small fraction of pair-wise distances that get computed are selected by:

computing sufficient pair-wise distances between training examples in each class to form rough intra-class clusters based on proximity;

assigning all remaining motion signals to a closest cluster in its class;

choosing a cluster centroid for each cluster and computing all pair-wise distances between cluster centroids;

using triangle inequality and the already computed pair-wise distances to approximate distances between examples that have not been computed;

computing additional pair-wise distances for edge cases where the triangle inequality breaks down.

48. A method for creating motion recognizers, the method comprising:

receiving a motion recognizer built from a training set composed of a first set of motion signals characterizing at least one type of motion executed over some period of time with a motion sensing device, wherein the motion signals include sufficient information to compute position and orientation over time of the motion sensing device;

receiving a second set of motion signals from a second motion sensing device providing sufficient information to compute position and orientation over time of the second motion sensing device; and

performing motion recognition to determine a first example motion signal in the training set most responsive to a second example in the second set of motion signals;

computing at any point in time a first 3D track of the first example motion signal, and a second 3D track of the second example motion signal; and

rendering the first and second 3D tracks visually side by side, with at least a first major point of divergence between the two motions highlighted.

49. The method as recited in claim 48, further comprising processing the first and second example motion signals respectively so that only interesting portions of the first and second example motion signals are used in the motion recognition or for creating one or more of the motion recognizers, wherein the interesting portions include one or more of:

a relative magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a relative magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time; and

a relatively large overall time has passed

50. The method as recited in claim 48, further comprising:

updating the motion recognizers with one or more motions from the second set of motion signals if the classification distance from such a motion to a prototype in a class of one of the motion recognizers is within a predefined threshold.