Sensor fusion interface for multiple sensor input

• US 9,389,681 B2
• Filed: 12/19/2011
• Issued: 07/12/2016
• Est. Priority Date: 12/19/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sankar, Aditya, Portnoy, William Lawrence
• Primary Examiner(s): Liu, Hexing
• Application Number:
  US13/329,777
• Publication Number:
  US 20130159350A1
• Time in Patent Office: 1,667 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

The subject disclosure is directed towards a sensor fusion interface that enables interaction between one or more entities of a physical environment and a computerized device component. A plurality of sensor modules generate multiple sensor input data associated with one or more entities in an environment and store such data in a shared library in accordance with a uniform and common schema. The multiple sensor input data is refined until a certain level of accuracy is achieved. Using the sensor fusion interface, entity state data is extracted from the shared library and exposed to the computerized device component.

Claims

1. In a computing environment, a method performed at least in part on at least one processor, comprising:

processing requests received via a sensor fusion interface to enable interaction between one or more entities and an computerized device component, including;

accessing a shared library that is coupled to a plurality of sensor modules corresponding to at least two input modalities for the one or more entities, the shared library comprising a schematized version of multiple sensor input data to represent a cross-sensor synthesis;

transforming the schematized version of the multiple sensor input data into entity state data; and

performing at least one of communicating an event corresponding to a change in entity pose, generating control data corresponding to different points-in-time, or correlating a portion of the entity state data associated with an entity and another portion of the entity state data associated with another entity to determine entity interactions.

2. The method of claim 1 further comprising providing the computerized device component with access to the entity state data.

3. The method of claim 2, wherein providing the computerized device component with access to the entity state data further comprises communicating one or more commands for the computerized device component from the one or more entities.

4. The method of claim 2, wherein providing the computerized device component with access to the entity state data further comprises generating control data based on the change in entity pose.

5. The method of claim 1 further comprising determining a position and an orientation of a device being manipulated by the one or more entities.

6. The method of claim 1 further comprising removing unwanted perturbations in the multiple sensor input data.

7. The method of claim 1, wherein accessing the shared library further comprises configuring the plurality of sensor modules to refine the multiple sensor input data.

8. The method of claim 1 further comprising defining symbolic activity that translates into commands for the computerized device component.

9. The method of claim 1 further comprising updating the entity state data in response to entity movement.

10. The method of claim 1 further comprising producing an environment model associated with the one or more entities.

11. The method of claim 1 further comprising correlating different portions of the entity state data to determine entity interactions between two or more entities.

12. In a computing environment, a system, comprising, at least one processor, a memory communicatively coupled to the at least one processor and including components comprising:

an interaction mechanism configured to provide a sensor fusion interface to an computerized device component, wherein, for each entity in an environment, the sensor fusion interface is configured to expose entity state data to the computerized device component, and to instruct a common knowledge data component to update the entity state data in response to entity activity, wherein the common knowledge data component is configured to integrate multi-modal input data from a plurality of sensor modules according to a uniform schema, to refine the multi-modal input data until a level of accuracy is achieved, and to identify the entity activity based on the refined multi-modal input data, including performing at least one of communicating an event corresponding to a change in entity pose, generating control data corresponding to different points-in-time, or correlating a portion of the entity state data associated with an entity and another portion of the entity state data associated with another entity to determine entity interactions.

13. The system of claim 12, wherein the interaction mechanism is further configured to operate a computerized device in the physical environment based on the entity activity.

14. The system of claim 12, wherein the interaction mechanism is further configured to control a virtual character on a virtual environment that is generated by the computerized device component.

15. The system of claim 12, wherein the interaction mechanism is further configured to initialize the plurality of sensor modules in response to the computerized device component.

16. The system of claim 12, wherein the interaction mechanism is further configured to identify a conflict between two or more entities.

17. The system of claim 12, wherein the interaction mechanism is further configured to expose an event that indicates entity movement out of a range of a device.

18. The system of claim 12, wherein the plurality of sensor modules include a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device, or any combination of a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device.

19. One or more computer-readable storage media having computer-executable instructions, which when executed perform steps, comprising:

processing a request for entity state data associated with an entity in a physical environment;

selecting at least two input modes among a plurality of input modes for producing the entity state data based on configuration data from a computerized device component, including processing a type system to format sensor input data corresponding to the plurality of input modes;

fusing the sensor input data to generate the entity state data; and

performing at least one of communicating to the computerized device component an event corresponding to a change in entity pose, generating control data corresponding to different points-in-time, or correlating a portion of the entity state data associated with an entity and another portion of the entity state data associated with another entity to determine entity interactions.

20. The one or more computer-readable storage media of claim 19 having further computer-executable instructions comprising:

examining entity interactions associated with a plurality of entities; and

communicating events to the computerized device component indicating conflicts corresponding to the entity interactions within the physical environment.