Systems and methods for motion recognition using multiple sensing streams

• US 8,041,659 B2
• Filed: 09/17/2010
• Issued: 10/18/2011
• Est. Priority Date: 07/14/2006
• Status: Expired due to Fees

• Current Assignee: AiLive, Inc.
• Original Assignee: AiLive, Inc.
• Inventors: Tu, Xiaoyuan, KAWANO, Yoichiro, Musick, JR., Charles, Powers, III, William Robert, Reynolds, Stuart, Wilkinson, Dana, Wright, Ian, Yen, Wei
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): RIFKIN, BEN M
• Application Number:
  US12/885,301
• Publication Number:
  US 20110112996A1
• Time in Patent Office: 396 Days
• Field of Search: 706/62, 706/14
• US Class Current: 706/14
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 10/776 Validation; Performance evaluation

Abstract

Techniques for motion recognition using multiple data streams are disclosed. Multiple data streams from inertia sensors as well as non-inertial sensors are received to derive a motion recognition signal from motion recognizers. These motion recognizers are originally constructed from a training set of motion signals and may be updated with received multiple sensing signals. In one aspect, multiple data streams are converted to device-independent motion signals that are applied with the motion recognizers to provide a generalized motion recognition capability.

Claims

1. A method for motion recognition from multiple data streams, the method comprising:

receiving in the processing unit a motion signal with at least two component outputs from at least one motion sensitive device associated with a set of sensors including inertial sensors and non-inertial sensors;

performing motion recognition with at least one of motion recognizers by classifying the motion signal, wherein at least one of the motion recognizers are constructed, prior to receiving the at least two component outputs, by a training set of motion signals created by an end user without reference to a predefined set of allowed motions, the training set of motion signals including a first set of motion signals characterizing at least one type of motion executed over a period of time, substantially all parameters needed to create at least one of motion recognizers that are ad-hoc are determined automatically;

transforming the component outputs into device-independent motion signals by an operation including virtually changing locations of the sensors to corresponding locations of sensors used to generate the training set of motion signals; and

applying the motion recognizers to the device-independent motion signals to provide a generalized motion recognition capability.

2. The method as recited in claim 1, wherein the non-inertial sensors include (i) buttons, dpads, sticks or triggers on the motion sensitive device, (ii) one or more touch screen sensors on the motion sensitive device, (iii) one or more video cameras disposed in front or vicinity of the user to track body part location and orientation of the user, and (iv) one or more microphones.

3. The method as recited in claim 1, comprising:

generating a continuous recognition signal for the motion signal in response to one of the motion recognizers which includes a set of prototypes, wherein said generating a continuous recognition signal proceeds without fully utilizing one or more of the prototypes in the one of the motion recognizers.

4. The method as recited in claim 1, further comprising:

choosing prototypes based on time warp distances, and these prototypes to be used to create one or more new motion recognizers, and wherein said performing motion recognition with at least one of the motion recognizers by classifying the motion signal comprises classifying the prototypes by computing time warp distances to the prototypes in the one or more new motion recognizers responsive to the motion sensitive device.

5. The method as recited in claim 1, wherein the processing unit is configured to include a recognition runtime library associating an elasticity parameter with each of the component outputs, and a subsequent time warp-related score is responsive to the elasticity parameter.

6. The method as recited in claim 5, wherein the runtime library is configured to recognize simultaneous independent motions, simultaneous dependent motions, and static poses of the motion sensitive device.

7. The method as in claim 1, further comprising applying the motion recognizers to the motion signal to build generalized motion recognizers responsive to another motion sensitive device.

8. A method for motion recognition from multiple data streams, the method comprising:

loading in a processing unit a set of motion recognizers that are created in advance by an end user using at least one first motion sensitive device embedded with a first set of sensors producing a set of training;

receiving in the processing unit a motion signal with at least two component outputs from at least one second motion sensitive device embedded with a second set of sensors some of which are different from some of the first set of sensors;

converting the component outputs into device-independent motion signals by an operation including virtually changing locations of the second set of sensors to corresponding locations of the first set of sensors used to generate the training signals; and

applying the motion recognizers to the device-independent motion signals to provide a generalized motion recognition capability.

9. The method as recited in claim 8, further comprising determining a final motion recognition signal from the motion signal using the generalized motion recognition capability.

10. The method as recited in claim 8, wherein the generalized motion recognition capability generates a continuous recognition signal for the motion signal in response to one of the motion recognizers which includes a set of prototypes, wherein said generating a continuous recognition signal proceeds without fully utilizing one or more of the prototypes in the one of the motion recognizers.

11. The method as recited in claim 9, further comprising:

choosing prototypes based on time warp distances, and these prototypes to be used to create one or more new motion recognizers, and wherein said applying of the motion recognizers to the device-independent motion signals to provide a generalized motion recognition capability comprises classifying the prototypes by computing time warp distances to the prototypes in the one or more new motion recognizers responsive to the at least one second motion sensitive device.

12. The method as recited in claim 9, wherein the first motion sensitive device is different from the second motion sensitive device in terms of its ability to fully describe various human motions.

13. The method as recited in claim 12, wherein the processing unit includes an interface to facilitate registration of new motion sensitive devices to inform the processing unit of sufficient statistics of the new motion sensitive devices to be used with the generalized motion recognition capability.

14. The method as recited in claim 8, wherein there is a third motion sensitive device, the second and third motion sensitive devices are respectively being held by two users, and motions signals from the second and third motion sensitive devices are correlated so that the two users perform paired motions at similar times and in complementary ways.

15. A system for motion recognition from multiple data streams, the system comprising:

a processing unit loaded with a set of motion recognizers that are created in advance by an end user using at least one first motion sensitive device with a first set of sensors producing at least a first motion signal, the processing unit receiving at least a second motion signal from a second motion sensitive device with a second set of sensors producing inertial sensor signals and other sensor signals describing various motions made by the end user, the motion second motion signal from the second motion sensitive device including the inertial sensor and other sensor signals,

wherein the processing unit is configured to convert the inertial sensor and other sensor signals into device-independent motion signals by an operation including virtually changing locations of the second set of sensors to corresponding locations of the first set of sensors and further apply the motion recognizers to the motion second signal to provide a generalized motion recognition service responsive to the second motion sensitive device.

16. The system as recited in claim 15, wherein the processing unit is further configured to determine a final motion recognition signal from the second motion signal using the generalized motion recognition service.

17. The system as recited in claim 16, wherein the processing unit is further configured to transform each of the inertial sensor and other sensor signals into one or more of a pseudo-linear acceleration, a pseudo-angular velocity, or a pseudo-button signal.

18. The system as recited in claim 15, wherein the first motion sensitive device is different from the second motion sensitive device in terms of sensibility.

19. The system as recited in claim 15, wherein the processing unit includes an interface to facilitate registration of motion sensitive devices to inform the processing unit of sufficient statistics of new motion sensitive devices to be used with the generalized motion recognition service.

20. The system as recited in claim 15, wherein the generalized motion recognition service generates a continuous recognition signal for the second motion signal in response to one of the motion recognizers which includes a set of prototypes, wherein said generating a continuous recognition signal proceeds without fully utilizing one or more of the prototypes in the one of the motion recognizers.