Self-contained inertial navigation system for interactive control using movable controllers

• US 9,405,372 B2
• Filed: 01/25/2008
• Issued: 08/02/2016
• Est. Priority Date: 07/14/2006
• Status: Active Grant

• Current Assignee: Wei Yen
• Original Assignee: AiLive, Inc.
• Inventors: Reynolds, Stuart, Tu, Xiaoyuan, Funge, John, Bererton, Curt, Wright, Ian, Yen, Wei, Dobson, Daniel, Musick, JR., Charles, Powers, III, William Robert
• Primary Examiner(s): Harper, Tramar
• Assistant Examiner(s): Clarke, Jr., Robert T
• Application Number:
  US12/020,431
• Publication Number:
  US 20100113153A1
• Time in Patent Office: 3,112 Days
• Field of Search: 463/36, 463/37, 463/47, 345156-158, 345162-163, 345/169, 703/141, 703/145, 703150-154, 4 3/6, 4 3/7, 4 4/7, 1 3/6, 1 3/7, 1 4/7
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1031 using a wireless connection, e.g. Bluetooth, infrared connections
• CPC Class Codes: A63F 2300/1043 being characterized by constructional details
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A movable game controller for controlling aspects of a computer controlled game display with apparatus for determining the linear and angular motion of that movable controller. The apparatus includes a plurality of self-contained inertial sensors for sensing the tri-axial linear and tri-axial angular motion of the moving controller. Each sensor is mounted at a fixed linear position and orientation with respect to the others. The linear and angular motion of the controller is computed from the correlated motion sensor readings of each of the plurality of self-contained inertial sensors.

Claims

1. A method for determining motion of a controlling device interacting with a 3D virtual environment being displayed on a screen, the method comprising:

determining in a computing device a set of estimated errors associated with a time series of data sampled from sensor signals generated from a plurality of inertial sensors disposed in configuration relations known to each other and enclosed in the controlling device, the sensor signals being sufficient to determine motion of the controlling device in six degrees of freedom, wherein the controlling device is being operated by a user, and is at least one of;

handheld, wearable or attachable to the user;

extracting, using the sensor signals, three orthogonal readings of angular motion of the controlling device;

determining an estimated orientation of the controlling device by using the three orthogonal readings of angular motion and a previous estimated orientation of the controlling device;

extracting, using the sensor signals, readings of linear motion of the controlling device;

calculating an estimated linear motion of the controlling device using readings of linear motion and the estimated orientation;

calculating a trajectory of the controlling device by updating the previous estimate of orientation and a relative location using the estimated orientation, wherein the trajectory demonstrates the motion of the controlling device in six degrees of freedom; and

causing at least a corresponding object in the 3D virtual environment to move according to the trajectory.

2. The method as in claim 1, wherein the controlling device is a composite controller integrating a basic controller and an auxiliary controller to provide the sensor signals sufficient to determine the motion of the controlling device in six degrees of freedom.

3. The method as in claim 1, wherein the estimated errors are constrained in accordance with human physiology for the 3D virtual environment.

4. The method as in claim 1, wherein the trajectory is calculated with constraints to confine the estimated orientation and linear motions and determine whether the estimated orientation and linear motions are permitted by rules designed particularly to the 3D virtual environment.

5. The method as in claim 1, wherein the set of estimated errors relates to errors in the time series of data.

6. The method as in claim 1, wherein the 3D virtual environment is a video game.

7. The method as in claim 6, wherein said determining a set of estimated errors operates in substantially real-time relative to time-varying position and orientation of the controlling device.

8. The method as in claim 6, wherein said determining a set of estimated errors is responsive to a set of sensors other than the self-contained inertial sensors.

9. The method as in claim 8, wherein the sensors other than the self-contained inertial sensors include at least one of

an infra-red sensor,

an optical recognizer,

a sonic device,

an electromagnetic field sensor,

a device capable of determining information from triangulation, and

a GPS receiver.

10. The method as in claim 6, further comprising identifying one or more periods of time in which the controlling device is substantially quiescent.

11. The method as in claim 10, wherein said identifying one or more periods of time includes identifying subsequences of the time series of data in which observed acceleration of the controlling device is substantially attributed to gravity alone.

12. The method as in claim 10, wherein data for one or more periods of quiescence includes information sufficient to determine a direction of gravity.

13. The method as in claim 10, further comprising:

determining an orientation of the controlling device for a period of quiescence in response to the time series of data and another stream of data from at least one sensor other than a self-contained inertial sensor.

14. The method as in claim 13, wherein the another stream of data includes information responsive to a direct pointing device.

15. The method as in claim 10, wherein changes in the six components of position and orientation of the controlling device are only provided for a subset of the time series of data starting with a period of quiescence for the controlling device.

16. The method as in claim 6, further comprising:

maintaining the set of estimated errors for use by time-varying changes in the six components of position and orientation in response to one or more later time-varying changes in the six components of position and orientation of the controlling device.

17. The method as in claim 6, further comprising:

maintaining the set of estimated errors for use by time-varying changes in the six components of position and orientation in response to one or more earlier time-varying changes in the six components of position and orientation of the controlling device.

18. The method as in claim 6, further comprising:

presenting an animation in response to time-varying changes in the six components of position and orientation of the controlling device.

19. The method as in claim 6, wherein said determining a set of estimated errors and determining a set of estimates of time-varying changes are performed iteratively in response to receipt of further data in the time series of data.

20. The method as in claim 6, further comprising:

determining a set of re-estimated errors in response to the set of estimated errors and information obtained after the set of estimated errors; and

determining a revised set of estimates of configuration of the controlling device in response to the time series of data and the set of re-estimated errors.

21. The method as in claim 6, further comprising:

determining a tentative set of estimates of configuration of the controlling device,

wherein said determining a set of estimates is responsive to the tentative set of estimates.

22. The method as in claim 6, wherein the assumptions include information regarding constraints on various configurations of the controlling device.

23. The method as in claim 6, wherein the assumptions include information regarding constraints on various information appearing in the time series of data.

24. The method as in claim 6, wherein

estimating time varying changes in the six components of the orientation of the controlling device is responsive to a first computation,

estimating time-varying changes in the six components of position of that object is responsive to a second computation, and

the first computation is substantially independent of the second computation.

25. The method as in claim 6, wherein the constraints are responsive to domain knowledge about at least one of

a typical use of the controlling device

a typical location of use of the controlling device,

a biomechanical feature of a user of the controlling device, and

a motion recognition signal associated with that time series of data.

26. The method as in claim 6, wherein

the constraints are responsive to domain knowledge about a particular application of the controlling device;

and

domain knowledge about a particular application includes at least one of one or more typical motions of the controlling device within the particular application, and a set of unlikely motions of the controlling device within the particular application.

27. The method as in claim 26, wherein the particular application of the controlling device includes a fictional world.

28. The method as in claim 27, wherein the particular application of the controlling device includes a set of rules applicable to the fictional world.

29. The method as in claim 6, wherein additional information regarding the various configurations for the controlling device is provided by an associated application interactive with the controlling device.

30. The method as in claim 29, wherein the associated application provides bounds on one or more of allowable positions, and single or multiple derivatives thereof;

and one or more of allowable orientations, and single and multiple derivatives thereof.

31. The method as in claim 29, wherein the additional information includes one or more components of an initial position and orientation for the controlling device.

32. The method as in claim 29, wherein the additional information includes one or more components of position and orientation for that controlling device known at a selected period of time.

33. A method for determining motion of a controlling device in a computing device, the method comprising:

determining in the computing device a set of estimates of time-varying changes in one or more of components of position and orientation of the controlling device in response to a time series of data generated from a plurality of self-contained inertial sensors disposed in configuration relations known to each other and used to determine the motion of the controlling device in at least six degrees of freedom in a multi-dimensional space, the controlling device being located in a terrestrial frame of reference, substantially movable by a human motive force, and being at least one of;

handheld, wearable, attachable to a living creature, wherein said determining of the set of estimates of time-varying changes in one or more of components of position and orientation of the controlling device comprises;

extracting from the time series of data three orthogonal readings of angular motion of the controlling device;

determining an estimated orientation of the controlling device using the three orthogonal readings of angular motion and a previous estimated orientation of the controlling device;

extracting from the time series of data readings of linear motion of the controlling device;

calculating an estimated linear motion of the controlling device using readings of linear motion and the estimated orientation;

providing a model of a multi-dimensional virtual environment in response to the estimated linear motion and orientation;

calculating a point and direction of interest in the model; and

generating in the computing device a trajectory of the controlling device from the time series of data according to the estimated linear motion and orientation and a set of estimated errors associated with the time series of data, wherein the trajectory demonstrates the motion of the controlling device in six degrees of freedom.

34. The method as in claim 33, wherein the trajectory is generated with constraints to confine the estimated orientation and linear motion and determine whether the estimated orientation and linear motion are permitted in the 3D virtual environment.

35. The method as in claim 34, wherein

the time series of data includes information sufficient to track changes in all six components of position and orientation of an object in a three-dimensional space, and

the method further comprises determining a set of estimates of time-varying changes in all of the six components of position and orientation of the controlling device.

36. The method, as in claim 34, wherein the point and direction of interest are used to select a virtual object in the 3D virtual environment.

37. The method, as in claim 36, wherein the 3D virtual environment is a video game.