Identifying entities based on sensor data

• US 9,866,927 B2
• Filed: 04/22/2016
• Issued: 01/09/2018
• Est. Priority Date: 04/22/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ganjam, Kris, DeBarr, David Douglas, Jain, Navendu
• Primary Examiner(s): Dsouza, Adolf
• Application Number:
  US15/135,828
• Publication Number:
  US 20170311053A1
• Time in Patent Office: 627 Days
• Field of Search: None
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

Sensor data from multiple sensors associated with a user is received. The sensors may include sensors of a smart phone, and sensors associated with other devices such as fitness trackers, video game consoles, and cameras. The sensor data is processed to identify entities such as persons, locations, and objects that may be of interest to the user. A personal digital assistant application can present information related to the identified entities to the user, and can allow the user to perform various queries with respect to the identified entities, and previously identified entities. In addition, the identified entities can be used to trigger one or more rules including recording when and where a particular entity is identified, and generating an alert when a particular entity is identified.

Claims

1. A system for automatically identifying entities based on sensor data and for performing one or more actions based on the identified entities, the system comprising:

at least one computer; and

an entity engine adapted to;

store a plurality of rules, wherein each rule is associated with an entity and an action;

receive sensor data from a plurality of sensors;

identify a plurality of entities from the sensor data;

determine one or more rules of the plurality of rules that match an entity of the plurality of entities;

perform the action associated with at least one of the determined one or more rules;

determine an entity of interest of the plurality of entities by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest, wherein the scoring is based on at least one of user query data or global query data, wherein the user query data includes a history of queries associated with a user of the at least one computer, and wherein the global query data includes a history of queries associated with a plurality of users other than the user;

determine attributes associated with the entity of interest, wherein the attributes comprise descriptive information about the entity of interest; and

display the entity of interest and one or more of the determined attributes on a device.

2. The system of claim 1, wherein the plurality of sensors comprises a plurality of internal sensors and a plurality of external sensors.

3. The system of claim 2, wherein the plurality of external sensors is associated with one or more of a camera, a fitness tracker, a security system, or a video game console, and wherein the plurality of internal sensors comprises one or more of a microphone, an accelerometer, a camera, a proximity sensor, a pedometer, or a global positioning system (GPS).

4. The system of claim 1, wherein the entity engine is further adapted to store indicators of one or more of the entities of the plurality of identified entities in a log associated with the user of the at least one computer.

5. The system of claim 4, wherein each indicator is associated with a time when the indicated entity was identified and a location where the indicated entity was identified.

6. The system of claim 1, wherein the entity engine is further adapted to receive one or more of the plurality of rules.

7. The system of claim 1, wherein the entity engine is further adapted to determine the descriptive information about the entity of interest from at least one of global entity data or user entity data.

8. The system of claim 1, wherein the entity engine adapted to identify the plurality of entities from the sensor data comprises the entity engine adapted to identify the plurality of entities from the sensor data using the sensor data, one or more previously identified entities and previously received sensor data.

9. The system of claim 1, wherein the entity engine is implemented on one or both of the at least one computing device or a cloud computing platform.

10. A system for automatically identifying entities based on sensor data and for providing attributes based on the identified entities, the system comprising:

an entity engine adapted to;

receive sensor data from a plurality of sensors of a device;

identify a plurality of entities from the sensor data;

determine an entity of interest of the plurality of entities by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest, wherein the scoring is based on at least one of user query data or global query data, wherein the user query data includes a history of queries associated with a user of the device, and wherein the global query data includes a history of queries associated with a plurality of users other than the user;

determine attributes associated with the entity of interest, wherein the attributes comprise descriptive information about the entity of interest; and

display the entity of interest and one or more of the determined attributes on the device.

11. The system of claim 10, wherein the entity engine is further adapted to:

receive user input associated with the user of the device; and

determine the entity of interest of the plurality of entities based on the user input.

12. The system of claim 10, wherein the entity engine is further adapted to:

store a plurality of rules, wherein each rule is associated with an entity and an action;

determine one or more rules of the plurality of rules that match an entity of the plurality of entities; and

perform the action associated with at least one of the determined one or more rules.

13. The system of claim 12, wherein the action comprises one or more of generating an alarm, generating a notification, performing an internet search, performing a text-to-speech translation, finding a location, and performing a financial transaction.

14. The system of claim 10, wherein the device comprises one or more of a smart phone, a laptop, desktop, or a tablet computer, and the plurality of sensors comprise one or more of a microphone, an accelerometer, a camera, a proximity sensor, a pedometer, and a global positioning system (GPS).

15. A method for automatically identifying entities based on sensor data and for determining attributes associated with the identified entities, the method comprising:

collecting sensor data from one or more sensors of a plurality of sensors associated with a device by the device;

identifying a plurality of entities from the sensor data by the device;

receiving user input by the device;

determining one or more entities of interest from the plurality of entities based on the received user input by the device and by scoring each of the plurality of entities and selecting the entity with the highest score as the entity of interest, wherein the scoring is based on at least one of user query data or global query data, wherein the user query data includes a history of queries associated with a user of the device, and wherein the global query data includes a history of queries associated with a plurality of users other than the user;

determining a query based on the received user input by the device;

determining attributes associated with the one or more entities of interest based on the query by the device, wherein the attributes comprise descriptive information about the one or more entities of interest; and

displaying the one or more entities of interest and one or more attributes of the determined attributes on a display associated with the device by the device.

16. The method of claim 15, further comprising storing indicators of one or more of the entities of the plurality of identified entities in a log associated with the user of the device.

17. The method of claim 16, wherein each indicator is associated with a time when the indicated entity was identified and a location where the indicated entity was identified.

18. The method of claim 15, wherein identifying the plurality of entities from the sensor data comprises identifying the plurality of entities from the sensor data using the sensor data, one or more previously identified entities, and previously received sensor data.

19. The method of claim 15, further comprising:

determining one or more rules of a plurality of rules that match an entity of the plurality of entities; and

performing an action associated with each rule that is determined to match an entity of the plurality of entities.

20. The method of claim 15, wherein the query is a select, project, join, union, and aggregation (SPJUA) query.