Systems and methods for generalized motion recognition

• US 8,112,371 B1
• Filed: 05/07/2010
• Issued: 02/07/2012
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: AiLive, Inc.
• Original Assignee: AiLive, Inc.
• Inventors: Wright, Ian, Yen, Wei, Tu, Xiaoyuan, KAWANO, Yoichiro, Musick, JR., Charles, Powers, III, William Robert, Reynolds, Stuart, Wilkinson, Dana
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Rifkin, Ben
• Application Number:
  US12/775,487
• Time in Patent Office: 641 Days
• Field of Search: 706/46, 706/14
• US Class Current: 706/14
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Techniques for performing motion recognition using generalized motion recognition capability are disclosed. At least two types of sensors are used to generate sensor signals about a motion made by a user. The sensor signals are then transformed to device-independent motion signals that may be used to build generalized motion recognizers or used with one or more generalized motion recognizers responsive to the sensors. At least one of objects in a virtual interactive environment is responsive to a recognition signal determined from the motion signal and the generalized motion recognizers.

Claims

1. A system for controlling objects in a virtual interactive environment, the system comprising:

a motion sensitive device including at least two types of sensors configured to generate sensor signals about a motion made by a user, one of the at least two types of the sensors being inertial sensors;

a processing unit configured to access a set of motion recognizers, to receive a motion signal including the sensor signals from the motion sensitive device, and to transform each of the sensor signals into device-independent motion signals by an operation including;

relocating virtually the sensors to different locations respectively designed for at least some of the sensors;

transforming the sensor signals to appear as though the sensor signals were generated from sensors located at the respectively designed different locations; and

applying the motion recognizers to the device-independent motion signals to provide a generalized motion recognition capability for the motion sensitive device, wherein at least one of objects in the virtual interactive environment being displayed is responsive to a final recognition signal determined from the motion signal and the generalized motion recognition capability.

2. The system as recited in claim 1, wherein the device-independent motion signals including one or more of a pseudo-linear acceleration, a pseudo-angular velocity, or a pseudo-button.

3. The system as recited in claim 1, wherein the sensor signals are from inertial sensors in a hand-held device, and one or more of (i) buttons, dpads,

sticks and triggers on the hand-held device, (ii) touch screen sensors of the hand-held device, (iii) tracked features from video streams describing body part location and orientation information, and (iv) microphones.

4. The system as recited in claim 1, wherein the display, the processing unit and at least one type of the sensors all reside on the hand-held device that is responsive to the motion.

5. The system as recited in claim 1, wherein the user is not skilled in motion control design or implementation, a training set is created from the sensor signals and used to generate the generalized motion recognizers built by a motion recognizer builder residing on the processing unit.

6. The system as recited in claim 5, wherein the user made the motion without reference to a predefined set of allowed motions and/or without restriction on how to execute the motion.

24. The system as recited in claim 3 wherein the tracked features include one or more of positions and orientations of the hand-held device, tracked movement over time of one or more fingers of a hand, positions and orientations of a head over time, and an output of a facial expression recognition system.

25. The system as recited in claim 1, further comprising a display screen, the virtual interactive environment being displayed on the display screen providing an interface for one or more of the following activities:

a) internet browsing;

b) television channel surfing;

or c) playing games; and

the at least one of objects responsive to the final recognition signal corresponds to a 2D or 3D mouse cursor, in-game objects or characters, or virtual locations in a menu.

7. A system for controlling objects in a virtual interactive environment, the system comprising:

a processing unit loaded with a set of the motion recognizers that are created in advance using at least one first motion sensitive device including a set of sensors;

a second motion sensitive device configured to produce inertial sensor signals and other sensor signals describing various motions made by a user, said sensor signals produced by a different set of sensors in the second motion sensitive device from the sensors in the first motion sensitive device; and

the processing unit configured to receive a motion signal from the second motion sensitive device, the motion signal including the inertial sensor and other sensor signals, configured to convert the inertial sensor and other sensor signals into device-independent motion signals by relocating virtually the sensors in the second motion sensitive device to locations of the set of sensors of the at least one first motion sensitive device, wherein the device-independent motion signals are applied with generalized motion recognizers responsive to the second motion sensitive device, at least one of objects in the virtual interactive environment is responsive to a final recognition determined from the motion signal and the generalized motion recognizers.

8. The system as recited in claim 7, wherein the processing unit is enclosed in the second motion sensitive device that is equipped with the display screen.

9. The system as recited in claim 8, wherein the second motion sensitive device is a smart phone responsive to motions made by a user thereof.

10. The system as in claim 7, wherein the user is not skilled in motion control design or implementation, a training set is created from the sensor signals and used to generate the generalized motion recognizers built by a motion recognizer builder residing on the processing unit.

11. The system as recited in claim 10, wherein the user made the motion without reference to a predefined set of allowed motions and/or without restriction on how to execute the motion.

26. The system as recited in claim 8, wherein the display screen is part of a television.

12. A method for controlling objects in a virtual interactive environment, the method comprising:

accessing a set of motion recognizers created from training sets of labeled processed motion signals;

receiving a motion signal including two or more component outputs from a motion sensitive device including at least two or more types of sensors, wherein each of the component outputs describes a different component of a motion made by a user;

transforming the component outputs of the motion signal into device-independent motion signals that are used with generalized motion recognizers responsive to the motion sensitive device, wherein said transforming the component outputs of the motion signal into device-independent motion signals comprises;

relocating virtually the sensors to different locations respectively designed for at least some of the sensors;

transforming the sensor signals to appear as though the sensor signals were generated from the sensors located at the respectively designed different locations; and

computing a motion recognition signal from the device-independent motion signals and the generalized motion recognizers, wherein the motion recognition signal is responsive to one or more component outputs of the motion made by the user, and at least one of the objects in the virtual interactive environment is responsive to the final recognition.

13. The method as recited in claim 12, wherein the two or more component outputs are from:

inertial sensors in the motion sensitive device, and one or more of (i) buttons, dpads, sticks and triggers on motion sensitive device, (ii) touch screen sensors of motion sensitive device, (iii) tracked features from video streams describing body part location and orientation information, and (iv) microphones.

14. The method as recited in claim 12, wherein the device-independent motion signals include one or more of a pseudo-linear acceleration, a pseudo-angular velocity, or a pseudo-button.

15. The method as in claim 12, further comprising executing a recognition runtime library associating an elasticity parameter with each of the two or more component outputs, and configuring a subsequent time warp-related score to be responsive to the elasticity parameter.

16. The method as in claim 12, further comprising filtering adaptively the motion signal with an adaptor specific to each component of the motion signal, so that some components are rarely filtered or modified, and others are frequently filtered or modified.

17. The method as in claim 16, wherein both the generalized motion recognizers and the motion recognition signal are responsive to both static poses and dynamic motions.

27. The method as recited in claim 15 wherein the tracked features include one or more of positions and orientations of the motion sensitive device, the tracked movement over time of one or more fingers of a hand, positions and orientations of a head over time, and an output of a facial expression recognition system.

28. The method as recited in claim 14, wherein the virtual interactive environment is displayed on a television, the virtual interactive environment provides an interface for one or more of the following activities:

a) internet browsing;

b) television channel surfing;

or c) playing games; and

the objects responsive to a final recognition signal correspond to a 2D or 3D mouse cursor, in-game objects or characters, or virtual locations in a menu.

18. A method for controlling objects in a virtual interactive environment, the method comprising:

loading in a processing unit a set of the motion recognizers that are created in advance using at least one first motion sensitive device embedded with a first set of sensors;

receiving in the processing unit a motion signal with at least two component outputs from a second motion sensitive device embedded with a second set of sensors some of which are different from some of the first set of sensors;

converting the component outputs into device-independent motion signals by relocating virtually some of the second set of sensors in the second motion sensitive device to locations of some of the first set of sensors of the first motion sensitive device;

applying the device-independent motion signals to a set of generalized motion recognizers; and

causing at least one of the objects in the virtual interactive environment to be responsive to a recognition signal determined from the motion signal and the generalized motion recognizers.

19. The method as recited in claim 18, wherein the processing unit is enclosed in the second motion sensitive device that is equipped with a display screen to display the virtual interactive environment.

20. The method as recited in claim 19, wherein the second motion sensitive device is a smart phone responsive to motions made by a user thereof.

21. The method as recited in claim 18, wherein the processing unit and the second motion sensitive device are separate devices, the motion signal is received wirelessly from the second motion sensitive device to the processing unit.

22. The method as in claim 18, wherein the user is not skilled in motion control design or implementation, a training set is created from device-independent motion signals and used to generate the generalized motion recognizers built by a motion recognizer builder residing on the processing unit.

23. The system as recited in claim 22, wherein the user made the motion without reference to a predefined set of allowed motions and/or without restriction on how to execute the motion.