Sensor fusion to combine sensor input data from multiple devices into one input stream

• US 9,662,569 B2
• Filed: 12/31/2008
• Issued: 05/30/2017
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Pering, Trevor, Rosario, Barbara, Want, Roy, Lyons, Kenton M.
• Primary Examiner(s): YOO, JASSON H
• Application Number:
  US12/347,949
• Publication Number:
  US 20100271302A1
• Time in Patent Office: 3,072 Days
• Field of Search: 463 7, 463 36, 463 37, 463 39
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/335 using Internet
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 2203/0384 Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device

Abstract

Software gaming applications may be written to accept an input from one or more standard controllers such a joystick with input buttons. However, multiple wearable sensors or hand held MID devices may enhance the gaming experience. These sensors may include devices such as accelerometers to detect movement of the extremity they are attached to and wirelessly communicate this information to a receiver. Embodiments are directed to using sensor fusion to combine sensor data from multiple wireless input devices, such as wearable sensors and MID devices together to form one logical input stream that is presented to an application and which the application sees are a standard controller.

Claims

1. An apparatus, comprising:

a processor;

a hardware interface to receive a plurality of inputs from a plurality of input devices that are to generate sensor data not recognized by a software application as originating from a control device with which the software application is configured to operate, wherein at least one of the plurality of input devices is a wireless wearable device that includes at least one sensor, and wherein the sensor data is to represent movements associated with different parts of a user'"'"'s body; and

a non-transitory computer readable storage medium comprising a set of stored instructions which, if executed, cause the processor to fuse the sensor data from the plurality of input devices into a unified data stream that emulates sensor data originating from a single control device with which the software application is configured to operate, wherein the set of stored instructions comprises a gaming application.

2. The apparatus as recited in claim 1, further comprising:

a plurality of wearable sensors designed to be worn on various parts of a user'"'"'s body, wherein the plurality of wearable sensors are to provide at least one of the plurality of inputs.

3. The apparatus as recited in claim 2, wherein at least one of the plurality of inputs is to be provided by a wireless handheld device that includes a motion sensor.

4. The apparatus as recited in claim 1, wherein at least one of the plurality of inputs is to be provided by a wireless handheld device that includes a motion sensor.

5. The apparatus from claim 1, where the unified data stream is represented to the software component as originating from a physical controller including one of a joystick, keyboard, mouse, or dance pad.

6. The apparatus as recited in claim 1, wherein at least one of the plurality of input devices from which the hardware interface is to receive input is a camera.

7. The apparatus of claim 1, wherein at least two of the plurality of input devices are wireless hand-held devices.

8. The apparatus of claim 1, wherein at least one input from the plurality of input devices is to be provided by a wireless phone.

9. A method, comprising:

receiving a plurality of inputs from a plurality of input devices that generate sensor data not recognized by a software application as originating from a control device with which the software application is configured to operate, wherein at least one of the plurality of input devices is a wireless wearable device that includes at least one sensor, and wherein the software application is a gaming application; and

fusing the sensor data from the plurality of input devices into a unified data stream that emulates sensor data originating from a single control device with which the software application is configured to operate.

10. The method as recited in claim 9, wherein the single control device is a joystick.

11. The method as recited in claim 9, wherein at least one of the plurality of inputs is from one of a plurality of wearable sensors designed to be worn on a user'"'"'s body.

12. The method as recited in claim 11, further including a plurality of wireless wearable devices that each includes at least one sensor.

13. The method as recited in claim 11, wherein at least one of the plurality of inputs is received from one or more of an accelerometer or a camera.

14. The method as recited in claim 9, where the unified data stream is represented to the software application as originating from a physical controller including one of a joystick, keyboard, mouse, or dance pad.

15. A system, comprising:

a computer including a stored gaming application, wherein the gaming application is to recognize a data stream from a joystick controller;

a hardware interface to receive a plurality of inputs from a plurality of input devices that are to generate sensor data not recognized by the gaming application as originating from a joystick controller with which the gaming application is configured to operate, wherein at least one of the plurality of input devices is a wireless wearable device that includes at least one sensor, and wherein the sensor data is to represent movements associated with different parts of a user'"'"'s body; and

a component to combine the sensor data from the plurality of input devices into a single data stream that emulates sensor data originating from a joystick controller presented to the gaming application.

16. The system as recited in claim 15, wherein at least one of the plurality of inputs is to be received from wearable sensors designed to be worn on various parts of a user'"'"'s body.

17. The system as recited in claim 16, wherein at least one of the plurality of inputs is to be received from a wireless wearable sensor.

18. The system as recited in claim 15, wherein at least one of the plurality of inputs is to be received from a mobile internet device (MID).

19. The system as recited in claim 18, wherein at least one of the plurality of inputs is to be received from an accelerometer.