Machine intelligent predictive communications and control system

• US 10,748,037 B2
• Filed: 01/29/2019
• Issued: 08/18/2020
• Est. Priority Date: 05/31/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Behringer, Adam Edwin, Lee, Johnny H., Bader, Stephen David, Wang, Hsiao-Lan
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US16/260,584
• Publication Number:
  US 20190156158A1
• Time in Patent Office: 567 Days
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A machine intelligent communication and control system is able to dynamically predict and adapt information presented to various users. The system provides a personalized experience with its ability to identify users, become contextually aware of the user'"'"'s location and environment, identify objects and actions, and present customized information tailored for the user and the current environment. The system is further able to control connected items within the environment based upon various user preference considerations.

Claims

1. A system for providing predictive information based on complexity and a knowledge level associated with a context, the system comprising:

an imaging sensor;

an audio sensor;

a display;

one or more computing devices, wherein the one or more computing devices are in communication with the imaging sensor, the audio sensor, and the display, the one or more computing devices including one or more processors, a memory comprising instructions that, when executed by the one or more processors, perform operations of;

receiving, by a contextual server, a set of input data, wherein the set of input data comprises at least one of;

imaging data captured by the imaging sensor, and

audio data captured by the audio sensor;

determining, by the contextual server, the context based on the set of input data, wherein the context includes;

an operator;

a location data of the operator;

an action taken by the operator; and

an object, wherein the operator interacts with the object through the action;

determining, by the contextual server, a goal of the operator based on the context, wherein the goal includes at least one intended action upon an intended object by the operator based on an operator profile database;

identifying, by the contextual server, a deviation of the context from the goal;

generating, by the contextual server based on the deviation, the predictive information, wherein the predictive information comprises at least one of an advice, a reminder, a warning, and an instruction;

determining, by the contextual server based on the context and the operator profile database, a level of complexity of the predictive information, wherein the level of complexity matches with the knowledge level of the operator about a subject matter of the context;

determining, by the contextual server based at least upon the location data, an output device; and

providing, by the contextual server, the predictive information through the output device according to the determined level of complexity of the predictive information.

2. The system of claim 1, wherein the imaging data comprises two or more frames, and the operations further comprise determining, based at least in part upon comparing the two or more frames of the imaging data, actions taking place within the field of view of the imaging sensor.

3. The system of claim 1, wherein the predictive information to be presented to the operator comprises instructions related to the actions taking place within the field of view of the imaging sensor.

4. The system of claim 1, wherein the one or more processors further perform operations of updating the operator profile database associated with the operator based at least in part upon the predictive information to be presented to the operator.

5. The system of claim 1, the operations further comprises:

determining by the contextual server based on the set of input data, a timing of providing the predictive information; and

delaying the providing of the predictive information until the determined timing arrives.

6. The system of claim 1, further comprising auxiliary devices, and the one or more processors further perform operations to send control instructions to one or more of the auxiliary devices.

7. A computer-implemented method for providing predictive information based on complexity and a knowledge level associated with a context, the method comprising:

receiving, by the contextual server, a set of input data, wherein the set of input data comprising at least one of;

imaging data, and

audio data;

determining, by the contextual server, the context based on the set of input data, wherein the context includes;

an identity of an operator;

a location data of the operator,

an action taken by the operator; and

an object, wherein the operator interacts with the object through the action;

determining, by the contextual server, a goal of the operator based on the context, wherein the goal includes at least one intended action upon an intended object by the operator based on an operator profile database;

generating, by the contextual server based on the context and the goal, the predictive information, wherein the predictive information comprises at least one of an advice, a reminder, a warning, and an instruction;

determining, by the contextual server based on the context and the operator profile database, a level of complexity of the predictive information, wherein the level of complexity matches with the knowledge level of the operator about a subject matter of the context;

modifying, by the contextual server, the predictive information according to the determined on the level of complexity; and

providing, by the contextual server, the modified predictive information through the output device.

8. The computer-implemented method of claim 7, further comprising sending, by the contextual server, instructions to one or more objects within an area to control the one or more objects.

9. The computer-implemented method of claim 7, wherein providing the modified predictive information is based at least in part upon the location data of the operator.

10. The computer-implemented method of claim 7, wherein determining the context is based at least in part upon a facial expression of the operator.

11. The computer-implemented method of claim 7, further comprising updating, by the contextual server, a skill level or the knowledge level of the operator based at least in part upon the predictive information.

12. The computer-implemented method of claim 7, wherein the set of input data further comprises health information of the object when a sensor associated with the set of input data detects the object as living matter.

13. The computer-implemented method of claim 7, further comprising determining when the location data of the operator matches with a location of the set of input data;

and providing of the modified predictive information when the operator is at the location of the set of input data.

14. The method as in claim 7, wherein the modifying of the predictive information reduces the complexity of the predictive information based at least in part upon the operator profile database associated with the operator.

15. The method as in claim 7, wherein the predictive information has a first data format, and the modified predictive information has a second data format that is distinct from the first format.

16. A system for providing predictive information based on complexity and knowledge levels associated with a context, the system comprising:

one or more sensors;

one or more display devices; and

one or more computing devices comprising memory, one or more processors, and one or more programming instructions stored in the memory and executable by the one or more processors to perform operations comprising;

receiving, by a contextual server, a set of input data based on the one or more sensors, wherein the set of input data comprises at least one of;

imaging data of an area captured by the imaging sensor, and

audio data of the area captured by the audio sensor;

determining, by the context server, a context of the operator and a goal of the operator, wherein the context includes at least one of;

an operator,

a location of the operator,

an action taken by the operator,

the location data corresponding to the area, and

one or more objects located within the area,

and the goal includes at least an intended action upon the one or more objects by the operator based on the user profile database, wherein the user profile database includes at least one of;

one or more calendar data,

preference data,

a level of skill or knowledge of the operator;

identifying, by the contextual server, a deviation of the context from the goal;

generating, by the contextual server based on the deviation, the predictive information; and

controlling the one or more objects according to the predictive information.

17. The system of claim 16, wherein the operations further comprising:

determining, by the contextual server based on the context and the operator profile database, a level of complexity of the predictive information, wherein the level of complexity matches with the knowledge level of the operator about a subject matter of the context;

generating, by the contextual server based on the deviation, the predictive information, wherein the predictive information comprises at least one of an advice, a reminder, a warning, and an instruction; and

providing, by the contextual server, the predictive information through an output device according to the determined level of complexity of the predictive information.

18. The system of claim 17, wherein the operations further comprise transliterating the predictive information from a first level of detail to a second level of detail, wherein the second level of detail is more detailed than the first level of detail.

19. The system of claim 16, wherein the operations further comprise determining, by the contextual server based at least upon the location data, an output device.

20. The system of claim 16, wherein controlling the one or more objects comprises one or more of:

transmitting a message for displaying on the one or more objects when the one or more objects are capable of displaying the message;

changing a power switch of the one or more objects when the one or more objects are power-operated; and

changing settings of the one or more objects.