Contextual prediction of user words and user actions

• US 8,938,688 B2
• Filed: 04/21/2006
• Issued: 01/20/2015
• Est. Priority Date: 12/04/1998
• Status: Active Grant

• Current Assignee: Cerence Operating Company (Cerence Inc.)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Bradford, Ethan R., Kay, David J., Longe, Michael, Van Meurs, Pim, Tandon, Gaurav
• Primary Examiner(s): Pitaro, Ryan
• Assistant Examiner(s): Pan, Yongjia
• Application Number:
  US11/379,747
• Publication Number:
  US 20060247915A1
• Time in Patent Office: 3,196 Days
• Field of Search: 715/810
• US Class Current: 715/789
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06F 9/45512 Command shells
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: H04M 1/72451 according to schedules, e.g. using calendar applications
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 1/72457 according to geographic location
• CPC Class Codes: H04M 2250/10 including a GPS signal receiver

Abstract

An operating sequence for a handheld computing device manages the device to automatically detect cues describing the device'"'"'s environmental and user actions performed with the device, learn which cues and cue combinations are relevant to predict user actions, and then in response to occurrence of the relevant cues, predictively implementing the appropriate user action or configuring the device in anticipation of user action.

Claims

1. A handheld computing device, comprising:

a user interface;

a digital data processor coupled to the user interface and programmed to perform operations comprising;

over time, monitoring and recording cues describing the device environment and monitoring and recording user performed actions of configuring the device and operating its application programs;

learning which if any of the recorded cues and combinations of recorded cues are relevant predictors of which of the recorded user actions;

for each user action having a learned relevant predictor, planning one or more of the following types of machine-executed actions;

configuring the device to streamline user performance of the user action;

causing the device to perform the user action;

responsive to occurrence of learned relevant predictors of user actions, performing the machine-executed actions planned for the learned relevant predictors that occurred;

the operations further comprising prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.

2. The device of claim 1, where the operations of monitoring and recording user performed actions of configuring the device and operating its application programs includes monitoring and recording activity including at least the following:

user performed actions of configuring hardware of the device;

user performed actions of starting an application program or system function;

user performed actions of operating menus including navigating said menus and selecting entries of said menus;

user performed actions of entering a hardware or software mode.

3. The device of claim 1, where the operation of configuring of the device to streamline user performance of the user action includes at least the following operations:

configuring hardware of the device;

starting an application program or system function of the device;

starting an operation whose initiation is represented upon one or more menus provided by the device;

bypassing or prioritizing one or more components of one or more menus provided by the device; and

entering a hardware or software mode of the device.

4. The device of claim 1, the cues comprising:

while in a messaging application, one or more of the following;

a message recipient, sender, subject, priority, delivery method, delivery channel;

time including one or more of the following;

day of week, month, year, time of day;

geographic location;

characteristics of ambient noise detected by the device;

software state of the device; and

application context of the device.

5. The device of claim 1, the operation of monitoring and recording cues describing environment of the device comprises operations of monitoring and recording at least the following:

physical state detected by one or more sensors;

application program state; and

changes in application program state.

6. The device of claim 1, where the planning operation comprises:

for a cue comprising arrival of a start time for an entry of an calendar application of the device, planning a configuration operation comprising invoking a silent mode of operation of the device;

for a cue comprising presence of a given character of ambient noise, planning one of the following configuration operations;

switching the device to a hands free mode, or increasing an output volume of the device;

for a cue comprising user selection of a prescribed menu entry, planning a configuration operation comprising configuring the menu to streamline an expected user-invoked follow up action;

for a cue comprising change in application state of a first application, planning a configuration operation comprising one of the following;

automatically opening a second application, streamlining opening of the second application, prioritizing access to functions offered by the second application.

7. The device of claim 1, where the monitoring and recording and learning operations are performed repeatedly.

8. The device of claim 1, the types of machine-executed actions further including:

prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the relevant predictor.

9. The device of claim 1, where the prioritizing operation comprises:

where the present environment of the device includes the device having a messaging application open, planning a disambiguation operation comprising promoting a user identifier as best interpretation of user-entered text input;

where the present environment of the device includes a detected chronological indication, planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with the detected chronological indication;

where the present environment of the device includes a software application being open with a user-positioned cursor residing in a form field, planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with a data type associated with the form field.

10. The device of claim 1, the operations further comprising:

responsive to receiving inherently ambiguous user-entered text input, prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with a present environment of the device, where said consistency is according to a predetermined logic independent of learned association between environment and interpretation of user-entered text input.

11. A handheld computing device, comprising:

first means for interfacing with a user;

coupled to the first means, second means for digital data processing including performing operations comprising;

over time, monitoring and recording cues describing the device environment and monitoring and recording user performed actions of configuring the device and operating its application programs;

learning which if any of the recorded cues and combinations of recorded cues are relevant predictors of which of the recorded user actions;

for each user action having a learned relevant predictor, planning one or more of the following types of machine-executed actions;

configuring the device to streamline user performance of the user action;

causing the device to perform the user action;

responsive to occurrence of learned relevant predictors of user actions, performing the machine-executed actions planned for the learned relevant predictors that occurred;

the operations further comprising prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.

12. An article of manufacture, comprising:

at least one non-transitory computer readable storage medium tangibly embodying a program of machine-readable instructions;

where said instructions are executable by a digital processing apparatus to perform operations to operate a computing device, said operations comprising;

over time, monitoring and recording cues describing the device environment and monitoring and recording user performed actions of configuring the device and operating its application programs;

learning which if any of the recorded cues and combinations of recorded cues are relevant predictors of which of the recorded user actions;

for each user action having a learned relevant predictor, planning one or more of the following types of machine-executed actions;

configuring the device to streamline user performance of the user action;

causing the device to perform the user action;

responsive to occurrence of learned relevant predictors of user actions, performing the machine-executed actions planned for the learned relevant predictors that occurred;

the operations further comprising prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.

13. An article of manufacture, comprising:

at least one non-transitory computer readable storage medium tangibly embodying a program of machine-readable instructions;

where said machine-readable instructions are executable by a digital processing apparatus to perform operations comprising;

altering programming of a handheld computing device to include added functionality, the added functionality including performance of machine-executed operations comprising;

over time, monitoring and recording cues describing the device environment and monitoring and recording user performed actions of configuring the device and operating its application programs;

learning which if any of the recorded cues and combinations of recorded cues are relevant predictors of which of the recorded user actions;

for each user action having a learned relevant predictor, planning one or more of the following types of machine-executed actions;

configuring the device to streamline user performance of the user action;

causing the device to perform the user action;

responsive to occurrence of learned relevant predictors of user actions, performing the machine-executed actions planned for the learned relevant predictors that occurred;

the operations further comprising prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.

14. An apparatus, comprising:

circuitry of multiple interconnected electrically conductive elements;

where said elements are electrically configured to perform operations to operate a computing device, the operations comprising;

over time, monitoring and recording cues describing a device environment and monitoring and recording user performed actions of configuring the device and operating its application programs;

learning which if any of the recorded cues and combinations of recorded cues are relevant predictors of which of the recorded user actions;

for each user action having a learned relevant predictor, planning one or more of the following types of machine-executed actions;

configuring the device to streamline user performance of the user action;

causing the device to perform the user action;

responsive to occurrence of learned relevant predictors of user actions, performing the machine-executed actions planned for the learned relevant predictors that occurred;

the operations further comprising prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with the present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.

15. A handheld computing device including a user interface and also including a digital data processor programmed to perform operations comprising:

detecting cues describing the device'"'"'s context including at least application program state and changes in application program state,

detecting user actions of configuring the device and operating application programs of the device,

learning which detected cues and cue combinations are relevant to which detected user actions,

in response to occurrence of the cues or cue combinations learned to be relevant to a given user action, executing at least part of the given user action or configuring the device in anticipation of the given user action; and

further comprising;

prioritizing interpretation of inherently ambiguous user-entered text input by favoring a disambiguated interpretation consistent with a present environment of the device, where the present environment of the device includes detected geographic location and the prioritizing operation comprises planning a disambiguation operation comprising favoring interpretations of user-entered text input consistent with said geographic location.