Systems and methods for utilizing personalized motion control in virtual environment

• US 9,050,528 B2
• Filed: 02/22/2010
• Issued: 06/09/2015
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Musick, JR., Charles, Reynolds, Stuart, Powers, III, William Robert, Wilkinson, Dana, Wright, Ian, Yen, Wei, KAWANO, Yoichiro, Tu, Xiaoyuan
• Primary Examiner(s): RIFKIN, BEN M
• Application Number:
  US12/709,523
• Publication Number:
  US 20110043443A1
• Time in Patent Office: 1,933 Days
• Field of Search: 706/46
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/2145 the surface being also a display device, e.g. touch screens
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/67 adaptively or by learning from player actions, e.g. skill level adjustment or by storing successful combat sequences for re-use
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6027 using adaptive systems learning from user actions, e.g. for skill level adjustment
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 18/40 Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06V 10/945 User interactive design; Environments; Toolboxes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques for controlling motions using motion recognizers generated in advance by users are described. According to embodiment, the motion recognizers created by end users are utilized to control virtual objects displayed in a virtual environment. By manipulating one or more motion sensitive devices, end users could command what the objects to do in the virtual environment. Motion signals from each of the motion sensitive devices are recognized in accordance with the motion recognizers created in advance by the users. One or more of the motion signals are at the same time utilized to tune the motion recognizers or create additional motion recognizers. As a result, the motion recognizers are constantly updated to be more accommodating to the user(s).

Claims

1. A system for controlling virtual objects in a display in a manner responsive to recognition of spontaneous motions of an end user interacting with a video game, the system comprising:

at least one motion sensing device sending motion signals responsive to the spontaneous motions; and

a processing unit configured to generate ad-hoc personalized motion recognizers from a set of training data before the end user starts to interact with the video game, wherein the set of training data is created from motion examples performed or chosen by the end user, the ad-hoc personalized motion recognizers are generated by computing a smallest possible subset of pairwise distances between all motion data in the training set, only pairwise distances between all different motions belonging to one class of the training set are computed and the processing unit is configured to receive the motion signals, compute motion recognition signals from the motion signals in reference to the ad-hoc personalized motion recognizers, recognize a new type of motion, when interacting with the video game, by creating a new motion recognizer when the new type of motion has been made and none of the ad-hoc motion recognizers are responsive to the new type of motion, and cause a display to show a virtual interactive environment in which movements of at least one of the virtual objects are responsive to the motion recognition signals, wherein the ad-hoc personalized motion recognizers are updated to include the new motion recognizer as the end user uses the motion sensing device to interact with the video game.

2. The system as recited in claim 1, where the motion sensing device includes inertial sensors responding to the spontaneous motions and generating the motion signals.

3. The system as recited in claim 2, wherein the display, the processing unit and the motion sensing device are all enclosed in single hand-held device responsive to the spontaneous motions.

4. The system as recited in claim 2, wherein the display, the processing unit and the motion sensing device are separate units, the processing unit receives the motion signals from the motion sensing device in a wireless or wired manner, and drives the display to show the virtual interactive environment.

5. The system as recited in claim 1, wherein the processing unit includes a recognition runtime library configured to generate the motion recognition signals and service connections with a motion sensitive application supporting the virtual interactive environment, and wherein the motion signals are independently and simultaneously processed in the processing unit in a manner responsive to one or more of the motion recognizers to respond to motion control-related queries from the motion sensitive application.

6. The system as recited in claim 5, wherein the processing unit further includes a recognizer maker configured to tune the motion recognizers with reference to one or more of the motion signals.

7. The system as recited in claim 6, wherein at least one of the motion recognizers is updated with the one or more of the motion signals if a classification distance from one of the spontaneous motions to a prototype in a class of the motion recognizer is within a predefined threshold.

8. The system as recited in claim 5, wherein the processing unit further includes a recognizer maker configured to generate new motion recognizers with reference to one or more of the motion signals.

9. The system as recited in claim 8, wherein classification tolerances are controlled by one or more slack parameters associated with the new motion recognizer so that the end user is able to control classification sensitivity without repeating the spontaneous motions.

10. The system as recited in claim 1 wherein at least one of the motion recognizers whose likelihood for returning an “

undetermined”

label for any one of the motion recognition signals is controlled by one or more of the following methods;

(i) adjusting the likelihood of returning undetermined by modifying a cost of distance during recognizer construction;

or (ii) creating an undetermined class and making it responsive to one or more examples labeled “

undetermined”

in a training set used to create the motion recognizers in advance.

11. The system as recited in claim 1, wherein the motion signals are processed with operation of

processing the motion signals so that only interesting portions of the motion signals are provided for computing the motion recognition signals, wherein the interesting portions of the motion signals include one or more of:

a distinct magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a distinct magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time; and

a distinct overall time has passed.

12. The system as recited in claim 11, wherein the processing unit further includes a motion start predictor configured to segment the motion signals incoming as a stream into separate motions which the processing unit then computes the motion recognition signals therefor.

13. The method as in claim 1, wherein remaining pairwise distances between the motion data in a class that cannot be accurately inferred are computed.

14. A method for controlling virtual objects in a video application in a manner responsive to recognition of spontaneous motions of an end user, the method comprising:

receiving motion signals from at least one motion sensing device when the end user uses the at least one motion sensing device to interact with a video application;

computing, in a processing unit, motion recognition signals from the motion signals with reference to personalized motion recognizers that are generated, before the end user starts to interact with the video application, from a set of training data, wherein the set of training data is created from motion examples performed or chosen by the end user, the personalized motion recognizers are generated by computing a smallest possible subset of pairwise distances between all motion data, only pairwise distances between all different motions belonging to one class of the training set are computed;

recognizing a new type of motion, when interacting with the video application, by creating a new motion recognizer when the new type of motion has been made and none of the ad-hoc motion recognizers are responsive to the new type of motion; and

causing at least one or more of the virtual objects to move in the video application in responsive to the motion recognition signals, wherein the personalized motion recognizers are updated to include the new motion recognizer as the end user uses the at least one motion sensing device to continue interacting with the video application.

15. The method as in claim 14, further comprising:

processing the motion signals so that only interesting portions of the motion signals are provided for computing the motion recognition signals, where the interesting portions of the motion signals include one or more of;

a distinct magnitude of linear or angular accelerations changes beyond a threshold from neighboring samples;

a distinct magnitude of one or more axes of accelerations has changed beyond a threshold over a predefined period of time; and

a distinct large overall time has passed.

16. The method as in claim 15, wherein the motion recognizers have a per-class slack parameter controlling a recognition tolerance which is computed when a corresponding one of the motion recognizers is created, the recognition tolerance is adjusted when the end user indicates that a given motion is harder, or easier to execute.

17. The method as in claim 14, wherein the processing unit further includes a recognizer maker configured to tune the motion recognizers with reference to one or more of the motion signals.

18. The method as recited in claim 17, wherein at least one of the motion recognizers is updated with the one or more of the motion signals if a classification distance from one of the spontaneous motions to a prototype in a class of the motion recognizer is within a predefined threshold.

19. The method as recited in claim 14, wherein the processing unit further includes a recognizer maker configured to generate new motion recognizers with reference to one or more of the motion signals.

20. The method as recited in claim 19, wherein classification tolerances are controlled by one or more slack parameters associated with the new motion recognizer so that the end user is able to control classification sensitivity without repeating the spontaneous motions.

21. The method as recited in claim 14, wherein the display, the processing unit and the motion sensing device are all enclosed in single hand-held device responsive to the spontaneous motions.

22. The method as recited in claim 14, wherein the display, the processing unit and the motion sensing device are separate units, the processing unit receives the motion signals from the motion sensing device in a wireless or wired manner, and drives the display to show the virtual interactive environment.

23. The method as recited in claim 14, wherein the motion recognizers have a capacity parameter controlling a number of distinct types of motions that a motion recognizer is able to recognize, the capacity parameter is computed when the motion recognizers are created, and is adjustable to cause the motion recognizer to add prototypes when the end user indicates that a recognition signal computed for a given motion signal is not undetermined.

24. The method as recited in claim 23, wherein a runtime recognition library in the processing unit is configured to return an undetermined recognition signal for a motion signal responsive to the spontaneous motions, where a motion sensitive application has a predetermined action that is responsive to the undetermined recognition signal that determines translations and/or rotations of one or more objects in a virtual environment being supported by the motion sensitive application.

25. The method as recited in claim 24 wherein the runtime recognition library uses a signal start predictor to segment the motion signals arriving as a stream into separate motions which the processing unit then computes the motion recognition signals therefor.

26. The method as recited in claim 24 wherein the motion recognition signals include a score that is responsive to how well the end user performed a move, the score is used to modify one or more objects in the virtual environment, wherein the score is computed by locking in on a specific prototype in the motion recognizers, and using the specific prototype to compute a score based on how far inside a classification distance to the move of the end user.