Sensor fusion interface for multiple sensor input

• US 10,409,836 B2
• Filed: 06/20/2016
• Issued: 09/10/2019
• Est. Priority Date: 12/19/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Sankar, Aditya, Portnoy, William Lawrence
• Primary Examiner(s): Uddin, Mohammed R
• Application Number:
  US15/186,772
• Publication Number:
  US 20160299959A1
• Time in Patent Office: 1,177 Days
• Field of Search: 707784, 707688, 707692, 707700, 707999101
• CPC Class Codes: G06F 16/213 with details for schema evolution support
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network

Abstract

The subject disclosure is directed towards a sensor fusion interface that enables interaction between one or more entities of a physical environment and a computerized device component. A plurality of sensor modules generate multiple sensor input data associated with one or more entities in an environment and store such data in a shared library in accordance with a uniform and common schema. The multiple sensor input data is refined until a certain level of accuracy is achieved. Using the sensor fusion interface, entity state data is extracted from the shared library and exposed to the computerized device component.

Claims

1. A method implemented on hardware comprising:

processing requests received via a sensor fusion interface to enable interaction between one or more entities and a computerized device component, including;

accessing a shared library that is coupled to a plurality of sensor modules corresponding to at least two input modalities for the one or more entities, the shared library comprising a schematized version of multiple sensor input data to represent a cross-sensor synthesis for correlating data corresponding to the at least two input modalities;

transforming the schematized version of the multiple sensor input data into entity state data wherein the entity state data comprises symbolic activities by the one or more entities;

generating control data for the computerized device component based on the entity state data; and

updating the entity state data in response to entity movement.

2. The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises communicating an event corresponding to a change in entity pose.

3. The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises communicating one or more commands for the computerized device component from the one or more entities.

4. The method of claim 1, wherein providing the computerized device component with access to the entity state data further comprises generating the control data based on the entity state data that corresponds to different points-in-time.

5. The method of claim 1, wherein extracting the entity state data further comprises determining a position and an orientation of a device being manipulated by the one or more entities.

6. The method of claim 1, wherein extracting the entity state data further comprises removing unwanted perturbations in the multiple sensor input data.

7. The method of claim 1, wherein accessing the shared library further comprises configuring the plurality of sensor modules to refine the multiple sensor input data.

8. The method of claim 1, wherein extracting the entity state data further comprises defining symbolic activity that translates into commands for the computerized device component.

9. The method of claim 1, wherein extracting the entity state data further comprises producing an environment model associated with the one or more entities.

10. The method of claim 1, wherein extracting the entity state data further comprises extracting a first portion of the entity state data associated with a first entity and a second portion of the entity state data associated with a second entity, and correlating the first portion with the second portion to determine entity interactions.

11. In a computing environment, a system comprising at least one processor and a memory communicatively coupled to the at least one processor and including components comprising:

an interaction mechanism that, when executed by the at least one processor, is configured to provide a sensor fusion interface to a computerized device component, wherein, for each entity in an environment, the sensor fusion interface, when executed by the at least one processor, is configured to expose a profile comprising entity state data comprising one or more symbolic activities by the entity to the computerized device component, and instruct a common knowledge data component to update the profile to include a current entity state in response to entity movement, wherein the common knowledge data component, when executed by the at least one processor, is configured to integrate multi-modal input data from a plurality of sensor modules according to a uniform schema for defining entity activities.

12. The system of claim 11, wherein the interaction mechanism, when executed by the at least one processor, is further configured to operate a computerized device in the environment based on the entity movement.

13. The system of claim 11, wherein the interaction mechanism, when executed by the at least one processor, is further configured to control a virtual character on a virtual environment that is generated by the computerized device component.

14. The system of claim 11, wherein the interaction mechanism, when executed by the at least one processor, is further configured to initialize the plurality of sensor modules.

15. The system of claim 11, wherein the interaction mechanism, when executed by the at least one processor, is further configured to identify a conflict between two or more entities.

16. The system of claim 11, wherein the interaction mechanism, when executed by the at least one processor, is further configured to expose an event that indicates entity movement out of a range of a device.

17. The system of claim 11, wherein the plurality of sensor modules include a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device, or any combination of a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone, or a pointing device.

18. A non-transitory computer-readable storage medium having computer-executable instructions, which when executed perform steps, comprising:

processing a request for entity state data associated with an entity in a physical environment wherein the entity state data comprises symbolic activities by the entity;

selecting at least two input modes among a plurality of input modes for producing the entity state data from a computerized device component, including processing a type system to format sensor input data corresponding to the at least two selected input modes;

fusing the sensor input data to generate the entity state data;

communicating the entity state data to the computerized device component;

identifying a conflict between additional entity state data corresponding to an additional entity and the entity state data associated with the entity; and

providing an indication of the conflict to the computerized device component to prompt the computerized device component to resolve the conflict.

19. The non-transitory computer-readable storage medium of claim 18, wherein identifying the conflict comprises:

examining entity interactions associated with a plurality of entities; and

detecting that a first event associated with the entity state data conflicts with a second event associated with the additional entity state data.

20. The non-transitory computer-readable storage medium of claim 18, wherein the sensor input data corresponding to the at least two selected input modes comprises sensor data detected via a plurality of sensor devices, the plurality of sensor devices including one or more of a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone or a pointing device, or any combination of a camera, a microphone, a digitizer, a motion processor, a gaming controller, a speech recognition module, a gesture recognition module, a facial recognition module, a mobile phone, or a pointing device.