BEHAVIOR RECOGNITION AND AUTOMATION USING A MOBILE DEVICE

• US 20160379105A1
• Filed: 06/24/2015
• Published: 12/29/2016
• Est. Priority Date: 06/24/2015
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Moore, Larry Richard Jr., Wang, Valerie, Sahasrabudhe, Sandeep
• Application Number:
  US14/748,912
• Publication Number:
  US 20160379105A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04L 67/53 using third party service providers
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/466 Learning process for intelligent management, e.g. learning user preferences for recommending movies services using the results of monitoring in broadcast systems H04H60/61

Abstract

Signals representing local events and/or state are captured at a mobile device and utilized by a machine learning system to recognize patterns of user behaviors and make predictions to automatically launch an application, initiate within-application activities, or perform other actions. The local signals may include, for example, location information such as geofence crossings; alarm settings; use of network connections like Wi-Fi, cellular, and Bluetooth®; device state such as battery level, charging status, and lock screen state; device movement indicating that the device user may be driving, walking, running, or stationary; audio routing such as headphones being used; telemetry data from other devices; and application state including launches and within-application activities. A feedback loop is supported in which the machine learning system may utilize feedback from the user as part of a learning process to adapt and tune the system'"'"'s predictions to improve the relevance of the predictions.

Claims

1. A mobile device, comprising:

one or more processors;

a user interface (UI) configured to interact with a user of the device using one of visual display or audio; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform an automated method for launching applications or initiating within-application activities, comprising

collecting signals representing events that are occurring locally on the device,

analyzing the collected signals to identify recurring patterns of sequences of events that result in an application launch or an initiation of one or more within-application activities,

using the recurring patterns to make a prediction of a future application launch or a future initiation of one or more within-application activities, and

automatically operating the device to launch an application or initiating one or more within-application activities responsively to the prediction.

2. The mobile device of claim 1 further including collecting at least a portion of the signals from a digital assistant that is supported on the device in which the digital assistant interacts with the user through the UI.

3. The mobile device of claim 1 further including performing the analyzing using one of Bayesian network, neural network, regression, or classification.

4. The mobile device of claim 1 in which the events include one or more of application launch events initiated by the user, within-application activity events initiated by the user, activity events including idle, stationary, walking, or running, driving events including vehicle telemetry, audio routing events including using an audio endpoint, geofence boundary crossing events, wireless network connection or disconnection events, short range network connection or disconnection events, battery charge state events, charger connection or disconnection events, alarm creation events, alarm deletion events, or lock state events.

5. The mobile device of claim 1 further including tracking a frequency of occurrences of events and launching the application or initiating the within-application activities responsively at least in part to the tracked frequency.

6. The mobile device of claim 1 further including using a digital assistant to support interactions with the user including participating in conversations and making suggestions for automated actions.

7. One or more computer-readable memories storing instructions which, when executed by one or more processors disposed in a device, implement a machine learning system adapted for:

receiving signals that are indicative of occurrences of events on the device;

creating an event history using the received signals, in which event history is represented using one or more tree structures including event occurrences by type that are populated into a probabilistic directed graph;

calculating a probability of an event using the event history; and

triggering an action responsively to the calculated probability.

8. The one or more computer-readable memories of claim 7 further including counting event occurrences in the one or more tree structures to generate a confidence level for the event probability and triggering the action, at least in part, responsively to confidence level.

9. The one or more computer-readable memories of claim 7 in which the action includes an automated application launch or an automated initiation of a within-application activity.

10. The one or more computer-readable memories of claim 7 further including making a request to a device user for confirmation of the action prior to the triggering.

11. The one or more computer-readable memories of claim 7 further including triggering a suggestion for an action and exposing the suggestion through a user interface to a device user.

12. The one or more computer-readable memories of claim 11 further including receiving user feedback to the suggestion.

13. The one or more computer-readable memories of claim 12 further including generating one or more tree structures in response to the user feedback.

14. A method for automating operations performed on an electronic device employed by a user, including:

capturing signals that represent occurrences of events that are local to the device over a time interval;

identifying one or more chains of events from the captured signals;

determining a probability that a chain of events leads to a launch of an application on the device by the user;

determining a level of confidence in the probability; and

automatically launching an application when the probability exceeds a predetermined probability threshold and the level exceeds a predetermined confidence threshold.

15. The method of claim 14 further including capturing within-application events that represent events or state associated with the application and determining a probability that a chain of events leads to an initiation of a within-application activity.

16. The method of claim 15 further including automatically initiating one or more within-application activities based on the probability that a chain of events leads to an initiation of a within-application activity.

17. The method of claim 14 further including supporting a digital assistant on the electronic device and utilizing the digital assistant to obtain a confirmation from the user prior to the automatic launching.

18. The method of claim 17 further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of interacting with the user, performing tasks, performing services, gathering information, operating the electronic device, or operating external devices.

19. The method of claim 14 further including exposing a user interface (UI) for collecting user feedback.

20. The method of claim 19 further including configuring a machine learning system to perform the identifying and probability determination and implementing a feedback loop to provide the user feedback to the machine learning system.