Customizing actions based on contextual data and voice-based inputs

• US 9,837,076 B1
• Filed: 12/18/2015
• Issued: 12/05/2017
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Stekkelpak, Zoltan, Simonyi, Gyula
• Primary Examiner(s): Pham, Thierry L
• Application Number:
  US14/975,479
• Time in Patent Office: 718 Days
• Field of Search: 704206, 704250, 704255, 704266, 704275
• CPC Class Codes: G10L 15/07 to the speaker
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Methods and systems are provided for customizing an action. In some implementations, voice input is received from a user and a context is determined from the voice input. Potential contextual data is identified based on the context and the voice input. A level of confidence is determined for an association of the potential contextual data and the context. An action is performed based on the voice input, the potential contextual data, and the level of confidence. The potential contextual data is used to customize the action.

Claims

1. A computer-implemented method comprising:

building an event model by identifying, from user data of one or more applications associated with a user, events related to the user;

receiving a voice input from the user that includes a command to perform a specific action, the action having a plurality of parameters, and each parameter specifying information necessary to perform the action;

determining whether the voice input includes sufficient information to perform the action;

in response to determining that the voice input does not include sufficient information to perform the action, identifying one or more missing parameters from the voice input that are required to perform the action;

identifying, using the event model, one or more current or future events related to the user that are relevant to the action, wherein each current or future event either is (i) currently occurring relative to a time when the voice input was received or (ii) will occur in the future relative to the time when the voice input was received;

customizing the action based on the one or more current or future events, comprising assigning values to one or more of the identified missing parameters of the action based on data associated with the one or more current or future events; and

performing the customized action in response to the voice input.

2. The method of claim 1, the event model further comprising:

a structured collection of data and relationships associated with the events, wherein the data is indicative of people, places, and times associated with the events.

3. The method of claim 1, further comprising:

identifying an action to be performed in response to the voice input by;

determining a context for the voice input that represents a subject of the voice input; and

determining that the action is associated with the context for the voice input.

4. The method of claim 3, further comprising:

processing the voice input and the context to determine that there is insufficient information to assign a value to one or more missing parameters of the plurality of parameters based only on the voice input and the context; and

assigning values to the one or more missing parameters based on data associated with the one or more current or future events.

5. The method of claim 1, wherein:

identifying, using the event model, one or more current or future events related to the user that are relevant to the action comprises;

identifying user data from the event model that is potentially relevant to the action;

determining a level of confidence for an association of one or more potentially relevant current or future events from the identified user data and the context of the voice input; and

identifying one or more current or future events related to the user that are relevant to the action from the potentially relevant events based on the determined level of confidence.

6. The method of claim 5, further comprising, when the level of confidence is sufficiently low, presenting a refining prompt to allow the user to select one or more possible values for the one or more missing parameters, the one or more possible values being determined based on data associated with the one or more current or future events.

7. The method of claim 6, further comprising, when the level of confidence is sufficiently high, assigning values to the one or more missing parameters based on data associated with the one or more current or future events without presenting the refining prompt.

8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

building an event model by identifying, from user data of one or more applications associated with a user, events related to the user;

receiving a voice input from the user that includes a command to perform a specific action, the action having a plurality of parameters, and each parameter specifying information necessary to perform the action;

determining whether the voice input includes sufficient information to perform the action;

in response to determining that the voice input does not include sufficient information to perform the action, identifying one or more missing parameters from the voice input required to perform the action;

identifying, using the event model, one or more current or future events related to the user that are relevant to the action, wherein each current or future event either is (i) currently occurring relative to a time when the voice input was received or (ii) will occur in the future relative to the time when the voice input was received;

customizing the action based on the one or more current or future events, comprising assigning values to one or more of the identified missing parameters of the action based on data associated with the one or more current or future events; and

performing the customized action in response to the voice input.

9. The system of claim 8, the event model further comprising:

comprises structured collection of data and relationships associated with current or future events, wherein the data is indicative of people, places, and times associated with the events.

10. The system of claim 8, the operations further comprising:

identifying an action to be performed in response to the voice input by;

determining a context for the voice input that represents a subject of the voice input; and

determining that the action is associated with the context for the voice input.

11. The system of claim 10, the operations further comprising:

processing the voice input and the context to determine that there is insufficient information to assign a value to one or more missing parameters of the plurality of parameters based only on the voice input and the context; and

assigning values to the one or more missing parameters based on data associated with the one or more current or future events.

12. The system of claim 8, wherein:

identifying, using the event model, one or more current or future events related to the user that are relevant to the action comprises;

identifying user data from the event model that is potentially relevant to the action;

determining a level of confidence for an association of the one or more potentially relevant current or future events from the identified user data and the context of the voice input; and

identifying one or more current or future events related to the user that are relevant to the action from the potentially relevant events based on the determined level of confidence.

13. The system of claim 12, the operations further comprising, when the level of confidence is sufficiently low, presenting a refining prompt to allow the user to select one or more possible values for the one or more missing parameters, the one or more possible values being determined based on data associated with the one or more current or future events.

14. The system of claim 13, the operations further comprising, when the level of confidence is sufficiently high, assigning values to the one or more missing parameters based on data associated with the one or more current or future events without presenting the refining prompt.

15. One or more non-transitory computer storage media encoded with a computer program, the computer program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

building an event model by identifying, from user data of one or more applications associated with a user, events related to the user;

receiving a voice input from the user that includes a command to perform a specific action, the action having a plurality of parameters, and each parameter specifying information necessary to perform the action;

determining whether the voice input includes sufficient information to perform the action;

in response to determining that the voice input does not include sufficient information to perform the action, identifying one or more missing parameters from the voice input required to perform the action;

identifying, using the event model, one or more current or future events related to the user that are relevant to the action, wherein each current or future event either is (i) currently occurring relative to a time when the voice input was received or (ii) will occur in the future relative to the time when the voice input was received;

customizing the action based on the one or more current or future events, comprising assigning values to one or more of the identified missing parameters of the action based on data associated with the one or more current or future events; and

performing the customized action in response to the voice input.

16. The non-transitory computer storage media of claim 15, the operations further comprising:

identifying an action to be performed in response to the voice input by;

determining a context for the voice input that represents a subject of the voice input; and

determining that the action is associated with the context for the voice input.

17. The non-transitory computer storage media of claim 15, the operations further comprising:

processing the voice input and the context to determine that there is insufficient information to assign a value to one or more missing parameters of the plurality of parameters based only on the voice input and the context; and

assigning values to the one or more missing parameters based on data associated with the one or more current or near-term events.

18. The non-transitory computer storage media of claim 15, wherein:

identifying, using the event model, one or more current or future events related to the user that are relevant to the action comprises;

identifying user data from the event model that is potentially relevant to the action;

determining a level of confidence for an association of the one or more potentially relevant current or future events from the identified user data and the context of the voice input; and

identifying one or more current or future events related to the user that are relevant to the action from the potentially relevant events based on the determined level of confidence.

19. The non-transitory computer storage media of claim 18, the operations further comprising, when the level of confidence is sufficiently low, presenting a refining prompt to allow the user to select one or more possible values for the one or more missing parameters, the one or more possible values being determined based on data associated with the one or more current or future events.

20. The non-transitory computer storage media of claim 19, the operations further comprising, when the level of confidence is sufficiently high, assigning values to the one or more missing parameters based on data associated with the one or more current or future events without presenting the refining prompt.