Systems and methods for motion recognition with minimum delay

• US 20110109548A1
• Filed: 09/03/2010
• Published: 05/12/2011
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: AiLive, Inc.
• Original Assignee: AiLive, Inc.
• Inventors: KAWANO, Yoichiro, Reynolds, Stuart, Tu, Xiaoyuan, Wright, Ian, Yen, Wei, Wilkinson, Dana, Musick, JR., Charles, Powers, III, William Robert
• Granted Patent: US 7,953,246 B1
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

Techniques for performing motion recognition with minimum delay are disclosed. A processing unit is provided to receive motion signals from at least one motion sensing device, where the motion signal describes motions made by a user. The processing unit is configured to access a set of prototypes included in a motion recognizer to generate corresponding recognition signals from the motion signals in response to the motion recognizer without considering one or more of the prototypes completely in the motion recognizer. Movements of at least one of the objects in a virtual interactive environment is responsive to the recognition signals such that feedback from the motions to control the one of the objects is immediate and substantially correct no matter how much of the motion signals have been received.

Claims

1. A system for controlling objects in a virtual interactive environment, the system comprising:

a processing unit, receiving a motion signal from at least one motion sensing device, configured to access a set of prototypes included in a motion recognizer to generate corresponding recognition signals from the motion signal in response to the motion recognizer without considering one or more of the prototypes completely in the motion recognizer, wherein the motion sensing device sends the motion signal responsive to motions made by a user, the recognition signals are continuously generated while the user is making the motions with the motion sensing device; and

wherein the processing unit causes a display to show the virtual interactive environment, movements of at least one of objects in the virtual interactive environment are responsive to the recognition signals such that feedback from the motions to control the one of the objects is immediate, no matter how much of the motion signal has been received, wherein the feedback is responsive to a measure of progress that has been made through a current best guess from the motions, and/or to a measure of confidence provided with the current best guess.

2. The system as recited in claim 1, wherein the processing unit, the display and the motion sensing device all reside on a single hand-held device that is responsive to the motions.

3. The system as recited in claim 1, wherein the motion signals responsive to the motions include component outputs from inertial sensors in the motion sensing device, and one or more of (i) buttons, dpads, sticks and triggers on the motion sensing device, (ii) one or more touch screen sensors, (iii) tracked features from video streams describing body part location and orientation information, and (iv) microphones.

4. The system as recited in claim 1, wherein the processing unit is configured to execute a recognition runtime library that computes an early cutoff test between the motion signal and each of the prototypes such that the each of the prototypes is eliminated out of the prototypes from further consideration for the motion signal when the early cutoff test succeeds.

5. The system as recited in claim 1, wherein the processing unit is configured to compute a ranked list of parameters, each of the parameters including a current progress of one of the prototypes being compared with the motion signal, current confidence in prediction of the one of the prototypes to the motion signal, and a current running distance of the one of the prototypes to the motion signal.

7. The system as recited in claim 1, wherein the processing unit is configured to use a disambiguation tree that organizes a collection of sets of moves that are commonly confused with each other at certain points in time, and identify a current node in the disambiguation tree that represents a set of moves that are currently confused with each other given how much of the motion signal has been received so far.

9. The system as recited in claim 1, wherein each of the prototypes is represented by a series of data frames, one of the recognition signals is generated from only an initial portion of the series of data frames so that the user feels that the one of the objects reacts almost immediately to the motions made by the user.

6. The system as recited in claim 6, wherein a final recognition signal corresponds to one of the prototypes that is determined from the ranked list of parameters.

8. The system as recited in claim 8, wherein the processing unit is further configured to allow at least one of the objects in the virtual interactive environment to move in response to a shared predesigned preamble of the set of moves that are currently confused with each other.

10. A method for controlling objects in a virtual interactive environment, the method comprising:

receiving a motion signal from at least one motion sensing device;

computing, in a processing unit, a motion recognition signal from the motion signal with reference to motion recognizers that are created in advance from a set of training data generated by an end user without reference to a predefined set of allowed motions and without restriction on how to execute motions; and

causing one of the objects in the virtual interactive environment to be responsive to the recognition signal which is continuously generated while the user is manipulating the motion sensing device, such that feedback from a motion to control the one of the objects is immediate no matter how much of the motion signal has been received, wherein the feedback is responsive to a measure of estimated progress and/or confidence in a current best guess motion of the user.

11. The method as recited in claim 10, wherein the motion signal responsive to the motions include component outputs from inertial sensors in the motion sensing device, and one or more of (i) buttons, dpads, sticks and triggers on the motion sensing device, (ii) one or more touch screen sensors, (iii) tracked features from video streams describing body part location and orientation information, and (iv) microphones.

12. The method as recited in claim 10, wherein said loading at least a motion recognizer and said generating a recognition signal from the motion signal take place in the motion-sensitive device, the motion-sensitive device includes a display screen to display the virtual interactive environment.

13. The method as recited in claim 10;

further comprising;

computing a ranked list of parameters, each of the parameters including a current progress of one of the prototypes being compared with the motion signal, current confidence in prediction of the one of the prototypes to the motion signal, and a current running distance of the one of the prototypes to the motion signal

14. The method as recited in claim 10, wherein the motion recognizer is embedded with a disambiguation tree that is provided to organize a collection of sets of moves that are commonly confused with each other at certain points in time, and the method further comprising:

identifying a current node in the disambiguation tree that represents a set of moves that are currently confused with each other given how much of the motion signal has been received so far; and

returning a set of currently confused moves to the application at any point in time before a final recognition signal to the motion signal is determined.

15. The method as recited in claim 14, further comprising causing at least one of the objects in the virtual interactive environment to move in response to a shared predesigned preamble of the set of moves that are currently confused with each other.

16. The method as recited in claim 10, further comprising:

executing a recognition runtime library that computes an early cutoff test between the motion signal and each of the prototypes such that the each of the prototypes is eliminated out of the prototypes from further consideration for the motion signal when the early cutoff succeeds.

17. The method as recited in claim 10, further comprising:

computing a ranked list of parameters, each of the parameters including a current progress of one of the prototypes being compared with the motion signal, current confidence in prediction of the one of the prototypes to the motion signal, and a current running distance of the one of the prototypes to the motion signal.

18. The method as recited in claim 17, wherein the recognition signal corresponds to one of the prototypes that is determined from the ranked list of parameters.

19. A system for controlling objects in a virtual interactive environment, the system comprising:

a motion sensing device sending out a motion signal responsive to motions made by a user, the motion signal being wirelessly received by a processing unit configured to access a set of prototypes included in a motion recognizer to generate corresponding recognition signals from the motion signal in response to the motion recognizer without considering one or more of the prototypes completely in the motion recognizer, wherein the recognition signals are continuously generated while the user is making the motions with the motion sensing device, and wherein the processing unit causes a display to show the virtual interactive environment, movements of at least one of objects in the virtual interactive environment are responsive to the recognition signals such that feedback from the motions to control the one of the objects is immediate, no matter how much of the motion signal has been received, wherein the feedback is responsive to a measure of progress that has been made through a current best guess from the motions, and/or to a measure of confidence provided with the current best guess.

20. The system as recited in claim 19, wherein the processing unit is further configured to execute a recognition runtime library that computes an early cutoff test between the motion signal and each of the prototypes such that the each of the prototypes is eliminated out of the prototypes from further consideration for the motion signal when the early cutoff succeeds.