Machine intelligent predictive communication and control system

• US 10,223,613 B2
• Filed: 05/31/2016
• Issued: 03/05/2019
• Est. Priority Date: 05/31/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Behringer, Adam Edwin, Lee, Johnny H, Bader, Stephen David, Wang, Hsiao-Lan
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US15/169,546
• Publication Number:
  US 20170344854A1
• Time in Patent Office: 1,008 Days
• CPC Class Codes: G06F 16/435 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata

Abstract

A machine intelligent communication and control system is able to dynamically predict and adapt information presented to various users. The system provides a personalized experience with its ability to identify users, become contextually aware of the user'"'"'s location and environment, identify objects and actions, and present customized information tailored for the user and the current environment. The system is further able to control connected items within the environment based upon various user preference considerations.

Claims

1. A system for providing predictive information based on complexity of an event and a knowledge level of an operator about the event, the system comprising:

an imaging sensor;

an audio sensor;

a display;

one or more computing devices in communication with the imaging sensors, the audio sensor, and the display, the one or more computing devices having memory, one or more processors, and one or more modules stored in the memory and executable by the one or more processors to perform operations to;

receive, by a contextual server, a set of input data, wherein the set of input data comprising at least one of;

imaging data from the imaging sensor, and

audio data from the audio sensor;

determine, by the contextual server based on the set of input data and an operator profile database, an operator;

determine, by the contextual server based on the set of input data and an object database, at least one object;

receive location data of the operator and the at least one object;

determine, by the contextual server based on the operator and the at least one object, a context of the operator and a goal of the operator, wherein the context comprises at least one action by the operator upon the at least one object according to the set of input data, and wherein the goal comprises at least one intended action upon an intended object by the operator based on the operator profile database;

identify, by the contextual server, a deviation of the context from the goal based on at least one difference between the at least one action and the at least one intended action;

generate, by the contextual server based at least in part upon the identified deviation, the predictive information, wherein the predictive information comprises at least one of an advice, a reminder, a warning, and an instruction;

determine, by the contextual server based at least in part upon the context and a profile associated with the operator, a level of complexity of the predictive information, wherein the level of complexity matches with the knowledge level of the operator about a subject matter of the context;

determine, by the contextual server based at least upon the location data, an output device for providing through which to present the predictive information;

transliterate, by the contextual server, the predictive information at the determined level of complexity and for the output device; and

provide, by the contextual server, the transliterated predictive information through the output device.

2. The system of claim 1, wherein the imaging data comprises two or more frames, and the operations further comprise determine, based at least in part upon comparing the two or more frames of the imaging data, actions taking place within the field of view of the imaging sensor.

3. The system of claim 2, the operation further comprising to:

wherein the predictive information to be presented to the operator comprises instructions related to the actions taking place within the field of view of the imaging sensor.

4. The system of claim 1, wherein the one or more processors further perform operations to update the profile associated with the operator based at least in part upon the predictive information to be presented to the operator.

5. The system of claim 1, wherein present the predictive information at the level of complexity through the output device is delayed based at least in part upon the location data indicating that the output device is moving.

6. The system of claim 1, further comprising auxiliary devices, and the one or more processors further perform operations to send control instructions to one or more of the auxiliary devices.

7. A computer-implemented method for providing information based on complexity of an event and a knowledge level of an operator about the event, the method comprising:

receiving, by a contextual server, a location of an operator, wherein the location corresponds to an area;

receiving, by the contextual server, video data of the operator within the area;

receiving, by a contextual server, audio data from the area;

determining, by the contextual server analyzing at least one of the video data or the audio data, an identity of the operator;

determining, by the contextual server analyzing the video data, one or more objects within the area;

determining, by the contextual server analyzing at least one of the video data or the audio data, actions by the operator occurring within the area;

determining, by the contextual server, information to be presented to the operator based at least in part on the actions occurring within the area in a first format, wherein the information comprises at least one of an advice, reminder, a warning, and an instruction;

retrieving, by the contextual server, a profile associated with the operator, the profile indicating at least one of a skill level and a knowledge level of the operator with respect to the actions occurring within the area;

transliterating, by the contextual server, the information to be provided to the operator from the first format to a second format based at least in part upon one of the skill level and the knowledge level; and

presenting, by the contextual server, the transliterated information to the operator in the second format.

8. The method as in claim 7, further comprising sending, by the contextual server, instructions to the one or more objects within the area to control the one or more objects.

9. The method as in claim 7, wherein presenting the information to the operator is based at least in part upon the location of the operator.

10. The method as in claim 7, wherein determining the information to be presented to the operator is based at least in part upon the facial expression of the operator.

11. The method as in claim 7, further comprising updating, by the contextual server, the skill level or the knowledge level of the operator based at least in part upon the information.

12. The method as in claim 7, further comprising receiving, la the contextual server from a sensor located within the area, health information related to a living thing within the area and wherein the information comprises the health information.

13. The method as in claim 7, wherein the information comprises a notification and the present the information to the operator is based at least in part on a determination that the operator is within the area.

14. The method as in claim 7, wherein the information is transliterated to reduce the complexity of the information based at least in part upon the profile associated with the operator.

15. The method as in claim 7, wherein the first format comprises at least one of video data, image data, a line drawing, voice data, and text data, and wherein the second format comprises a data format other than the first format.

16. A system for providing predictive information based on complexity of an event and knowledge levels of at operators about the event, the system, comprising:

one or more sensors;

one or more display devices; and

one or more computing devices comprising;

memory;

one or more processors; and

one or more modules stored in the memory and executable by the one or more processors to perform operations comprising;

determine, through data generated by the one or more sensors, an identity of a first operator;

determine, through the data generated by the one or more sensors, a location of the first operator, the location corresponding to an area;

determine, through the data generated by the one or more sensors, one or more objects located within the area;

receive a first profile associated with the first operator, the first profile comprising one or more of calendar data, preference data, a level of skill or knowledge of the first operator, and information associated with the first operator;

determine, based on the first operator and the one or more objects, a first context of the first operator and a first goal of the first operator, wherein the first context comprises at least an action by the first operator upon the one or more objects, and wherein the first goal comprises at least a first intended action upon an intended object by the first operator based on the first profile;

identify, by the contextual server, a deviation of the first context from the first goal based on at least one difference between the at least one action and the at least one intended action; and

control the one or more objects located within the area based at least in part on the at least one deviation of the first context from the first goal.

17. The system as in claim 16, wherein the operations further comprise:

determine, through data generated by the one or more sensors, an identity of a second operator;

receive a second profile associated with the second operator, the second profile comprising one or more of calendar data, preference data, a level of skill or knowledge of the second operator, and information associated with the second operator;

determine, based on the second operator and the one or more objects, a second context of the second operator and a second goal of the second operator, wherein the second context comprises at least an action by the second operator upon the one or more objects, and wherein the second goal comprises at least a second intended action upon an intended object by the second operator based on the second profile;

identify, by the contextual server, a deviation of the second context from the second goal based on at least one difference between the at least one action and the at least one intended action;

control the one or more objects located within the area based at least in part on the at least one deviation of the second context from the second goal;

determine, based at least in part upon the location, the one or more objects, and the first profile, a first predictive information to be provided to the first operator; and

determine, based at least in part upon the location, the one or more objects, and the second profile, a second predictive information to be provided to the second operator.

18. The system as in claim 17, wherein the operations further comprise transliterate the predictive information from a first level of detail to a second level of detail, wherein the second level of detail is more detailed than the first level of detail.

19. The system as in claim 16, wherein the operations further comprise determine a device upon which to present the information, and present the information on the device.

20. The system as in claim 16, wherein control the one or more objects comprises changing settings of the one or more objects.