Contextual entity resolution

• US 10,229,680 B1
• Filed: 12/29/2016
• Issued: 03/12/2019
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Gillespie, Kellen, Gens, Melanie Chie Bomke
• Primary Examiner(s): Neway, Samuel G
• Application Number:
  US15/394,753
• Time in Patent Office: 803 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Methods and systems for resolving entities using multi-modal functionality are described herein. Voice activated electronic devices may, in some embodiments, be capable of displaying content using a display screen. Contextual metadata representing the content rendered by the display screen may describe entities having similar attributes as an identified intent from natural language understanding processing. When natural language understanding processing attempts to resolve one or more declared slots for a particular intent, matching slots from the contextual metadata may be determined, and the matching entities may be placed in an intent selected context file to be included with the natural language understanding'"'"'s output data. The output data may be provided to a corresponding application for causing one or more actions to be performed.

Claims

1. A method, comprising:

receiving, at a speech-processing system, first audio data representing a first utterance, the first utterance being a request to perform a first action, and the first audio data being received from an electronic device;

receiving, at the speech-processing system, a first notification indicating that first content was presented, by a display screen of the electronic device, while at least a portion of the first utterance was spoken;

requesting, from a component of the speech-processing system, contextual data representing the first content;

determining that a first domain is responsible for providing the first content;

receiving first contextual data representing the first content displayed by the display screen;

generating first text data representing the first audio data by performing speech-to-text processing to the first audio data;

determining, using the first text data, that the first utterance corresponds to a first intent, the first intent being associated with the first domain;

determining that the first intent comprises at least a first slot;

determining, based at least in part on the first text data, a presence of an anaphoric term;

generating output data including the first intent and the first contextual data based on the anaphoric term being associated with the first contextual data; and

sending the output data to the first domain so as to cause;

a first value associated with the first slot to be determined based at least in part on the first contextual data, and

the first action to be executed using the first value.

2. The method of claim 1, wherein receiving the first contextual data further comprises:

receiving the first contextual data including at least one slot corresponding to the first intent and at least one value associated with the at least one slot such that the first domain is capable of using the output data to resolve an entity associated with the first slot.

3. The method of claim 1, further comprising:

determining that the first intent is a purchasing intent;

determining that the first slot is an item name slot; and

determining that the first contextual data comprises the item name slot and a value associated with the item name slot such that the output data includes the purchasing intent, the first contextual data, the item name slot, and the value associated with the item name slot.

4. The method of claim 1, further comprising:

receiving second audio data representing a second utterance, the second utterance being a different request to perform a second action, and the second audio data being received from the electronic device;

receiving a second notification indicating that second content was presented, by the display screen, while at least a portion of the second utterance was spoken;

requesting, from the component, additional contextual data representing the second content;

determining that the first domain is responsible for providing the second content;

receiving second entity data representing the second content;

generating second text data representing the second audio data by performing the speech-to-text processing to the second audio data;

determining, using the second text data, that the second utterance corresponds to a second intent to perform a second action associated with a second domain;

determining that the second intent comprises at least a second slot;

determining that the second contextual data is to be ignored based, at least in part, on the second contextual data corresponding to the first domain and the second utterance corresponding to the second domain;

generating additional output data including the second intent; and

sending the additional output data to the second domain.

5. A method, comprising:

receiving, from a first device, first audio data representing a first utterance;

receiving a first indication that first content was presented, by a display screen associated with the first device, while at least a portion of the first utterance was spoken;

determining a first domain that provided the first content;

receiving first contextual data representing the first content;

generating first text data representing the first audio data;

determining, based at least in part on the first text data, a first intent of the first utterance, the first intent being associated with the first domain, and the first intent comprising at least a first slot;

generating, based at least in part on the first text data, first output data comprising at least the first intent and the first contextual data; and

sending the first output data to the first domain so as to cause;

a first value associated with the first slot to be determined based at least in part on the first contextual data, and

at least one function associated with the first intent to be executed using the first value.

6. The method of claim 5, wherein determining the first intent further comprises:

determining that the first intent further comprises at least a second slot; and

determining, using the first text data, that a second value is associated with the second slot.

7. The method of claim 5, further comprising:

determining that the first contextual data comprises the first value associated with the first slot such that the first output data further comprises the first slot and the first value.

8. The method of claim 5, further comprising;

determining that the first intent is associated with an action to be performed to a list;

determining that the first slot corresponds to a list position slot; and

determining, based at least in part on the first contextual data, that a first list position value is associated with the list position slot.

9. The method of claim 5, further comprising:

determining that the first slot comprises an anaphoric term slot, the first output data being generated further based on the anaphoric term slot.

10. The method of claim 5, further comprising:

requesting, based at least in part on the first indication being received, contextual data from a plurality of domains including the first domain;

receiving a second indication from the first domain indicating that the first domain provided the first content to the first device.

11. The method of claim 5, further comprising:

receiving, from the first device, second audio data representing a second utterance;

receiving a second indication from the first device that that indicates an absence of content presented by the display screen;

generating second text data representing the second audio data;

determining, based at least in part on the second text data, a second intent of the second utterance, the second intent being associated with a second domain, and the second intent comprising at least a second slot;

generating, using the second text data, second output data comprising the second intent; and

sending the second output data to the second domain.

12. The method of claim 5, wherein generating the first output data further comprises:

determining that the first contextual data comprises a second slot;

determining, based at least in part on the second slot being equivalent to the first slot, a heuristics score indicating a number of same entity slots; and

determining that the heuristics score is greater than zero, the first output data further comprising the heuristics score.

13. The method of claim 5, further comprising:

receiving, from the first device, second audio data representing a second utterance;

receiving a second indication that second content was output, by the first device, while at least a portion of the second utterance was spoken;

determining the first domain provided the second content;

receiving second contextual data representing the second content;

generating second text data representing the second audio data;

determining, using the second text data, a second intent of the second utterance, the second intent being associated with a second domain, and the second intent comprising at least a second slot;

generating, based at least in part on the second text data, second output data comprising at least the second intent and the second contextual data; and

sending the second output data to the second domain.

14. A system, comprising:

at least one processor; and

at least one computer-readable medium encoded with instructions which, when executed by the at least one processor, cause the system to;

receive, from a first device, first audio data representing a first utterance;

receive a first indication from the first device that first content was presented, by a display screen associated with the first device, while at least a portion of the first utterance was spoken;

determine a first domain that provided the first content to the first device as at least a portion of the first utterance was spoken;

receive first contextual data representing the first content;

generate first text data representing the first audio data;

determine, based at least in part on the first text data, a first intent of the first utterance, the first intent being associated with the first domain, and the first intent comprising at least a first slot;

generate, based at least in part on the first text data, first output data comprising at least the first intent and the first contextual data; and

send the first output data to the first domain so as to cause;

a first value associated with the first slot to be determined based at least in part on the first contextual data, and

at least one function associated with the first intent to be executed using the first value.

15. The system of claim 14, wherein the at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

determine that the first intent further comprises at least a second slot; and

determine, using the first text data a second value associated with the second slot.

16. The system of claim 14, wherein the at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

determine that the first contextual data comprises the first value associated with the first slot such that the first output data further comprises the first slot and the first value.

17. The system of claim 14, wherein the at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

determine that the first intent is associated with an action to be performed to a list;

determine that the first slot corresponds to a list position slot; and

determine, based at least in part on the first contextual data, that a first list position value is associated with the list position slot.

18. The system of claim 14, wherein at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

determine that the first slot comprises an anaphoric term slot, the first output data being generated further based on the anaphoric term slot.

19. The system of claim 14, wherein the at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

request, based at least in part on the first indication being received, contextual data from a plurality of domains including the first domain;

receive a second indication from the first domain indicating that the first domain provided the first content to the first device.

20. The system of claim 14, wherein the at least one computer-readable medium is encoded with additional instruction which, when executed by the at least one processor, further cause the system to:

receive, from the first device, second audio data representing a second utterance;

receive a second indication from the first device that that indicates an absence of content being presented by the display screen;

generate second text data representing the second audio data;

determine, based at least in part on the second text data, a second intent of the second utterance, the second intent being associated with a second domain, and the second intent comprising at least a second slot;

generate, using the second text data, second output data comprising the second intent; and

send the second output data to the second domain.