Method of and system for real time feedback in an incremental speech input interface

• US 10,121,493 B2
• Filed: 05/07/2014
• Issued: 11/06/2018
• Est. Priority Date: 05/07/2013
• Status: Active Grant

• Current Assignee: Veveo, Inc. (Adeia Inc.)
• Original Assignee: Veveo, Inc. (Adeia Inc.)
• Inventors: Aravamudan, Murali, Welling, Girish, Gill, Daren, Ardhanari, Sankar, Barve, Rakesh, Venkataraman, Sashikumar
• Primary Examiner(s): Perveen, Rehana
• Assistant Examiner(s): Wong, Huen
• Application Number:
  US14/271,869
• Publication Number:
  US 20140337370A1
• Time in Patent Office: 1,644 Days
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/433 using audio data
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/54 for retrieval

Abstract

The present disclosure provides systems and methods for selecting and presenting content items based on user input. The method includes receiving first input intended to identify a desired content item among content items associated with metadata, determining that an input portion has an importance measure exceeding a threshold, and providing feedback identifying the input portion. The method further includes receiving second input, and inferring user intent to alter or supplement the first input with the second input. The method further includes, upon inferring intent to alter the first input, determining an alternative query by modifying the first input based on the second input, and, upon inferring intent to supplement the first input, determining an alternative query by combining the first input and the second input. The method further includes selecting and presenting a subset of content items based on comparing the alternative query and metadata associated with the subset.

Claims

1. A computer-implemented method for selecting and presenting content items based on user input, the method comprising:

providing access to a set of content items, the content items being associated with metadata that describes a corresponding content item;

receiving a first input intended by the user to identify at least one desired content item;

determining that a portion of the first input has a measure of importance that exceeds a threshold value;

providing feedback to the user identifying the portion of the first input;

receiving second input from the user that is subsequent to the first input;

determining a measure of similarity between the portion of the first input and the second input;

comparing the measure of similarity with a threshold;

upon a condition when the measure of similarity is determined to be above the threshold, determining that the portion of the first input is phonetically similar to the second input;

upon a condition when the measure of similarity is determined to not be above the threshold, determining that the portion of the first input is not phonetically similar to the second input;

based on determining that the portion of the first input is phonetically similar to the second input, determining an alternative query input by altering the first input based on the second input;

based on determining that the portion of the first input is not phonetically similar to the second input, determining the alternative query input by supplementing the first input with the second input;

selecting a subset of content items from the set of content items based on comparing the alternative query input and the metadata associated with the subset of content items; and

presenting the subset of content items to the user.

2. The method of claim 1,

wherein the determining that the portion of the first input has the measure of importance that exceeds the threshold value includes identifying one or more phrase boundaries in incremental input of the first input, and

wherein the identifying the one or more phrase boundaries is based at least in part on at least one of (a) an identified disfluency from the user in the first input, (b) grammar rules applied to the first input, (c) the importance measure of the portion of the first input, (d) at least one previous conversational interaction with the user, and (e) a user preference signature,

the user preference signature describing preferences of the user for at least one of (i) particular content items and (ii) particular metadata associated with the content items, wherein the portion of the first input is identified based on the user preference signature,

wherein the disfluency includes at least one of a pause in speech input, an auditory time filler in speech input, and a pause in typing input.

3. The method of claim 1, wherein selecting the subset of content items is based further upon a disfluency identified in the first input, and based further upon prior conversational interactions that are determined to be related to the first input and the second input.

4. The method of claim 1, wherein the providing the feedback includes at least one of:

requesting clarification on the identified portion of the first input,

suggesting a completion of the received first input, and

repeating the portion of the first input to the user, so as to notify the user that the portion of the first input is potentially incorrectly recognized.

5. The method of claim 4, wherein the requesting clarification on the identified portion of the first input is based at least in part on a determination that a disfluency occurs after the user has provided the portion of the first input.

6. The method of claim 4, wherein the suggesting the completion of the received first input is based at least in part on a determination that the disfluency occurs before the user would be expected to provide the portion of the first input.

7. The method of claim 1, wherein the feedback provided to the user is chosen based on at least one of:

a duration of an identified disfluency in the first input,

a measure of confidence in correct speech-to-text recognition of the portion of the first input,

a count of ambiguities detected in the first input,

a count of error corrections needed to identify the portion of the first input,

a count of nodes in a graph data structure, wherein the count of the nodes in the graph data structure measures a path between a first node representing an item of interest from a previous conversational interaction and a second node representing the portion of the first input, and

a measure of relatedness of the portion of the first input to previous conversational interactions with the user.

8. The method of claim 1, wherein the presenting the subset of content items includes presenting the subset of content items before receiving completed input from the user, upon a determination of a strong recognition match for the first input and upon a determination that a responsiveness measure of the selected subset of content items would be above a threshold.

17. The method of claim 1, wherein determining the measure of similarity between the portion of the first input and the portion of the second input includes detecting similar characters in the portion of the first input and the portion of the second input.

18. The method of claim 1, wherein determining the measure of similarity between the portion of the first input and the portion of the second input includes detecting similar phonetic variations in the portion of the first input and the portion of the second input.

9. A system for selecting and presenting content items based on user input, the system comprising:

computer readable instructions encoded on a non-transitory computer readable medium, the computer readable instructions causing the computer system to be configured to;

provide access to a set of content items, the content items being associated with metadata that describes a corresponding content item;

receive a first input intended by the user to identify at least one desired content item;

determine that a portion of the first input has a measure of importance that exceeds a threshold value;

provide feedback to the user identifying the portion of the first input;

receive second input from the user that is subsequent to the first input;

determine a measure of similarity between the portion of the first input and the second input;

compare the measure of similarity to with threshold;

upon a condition when the measure of similarity is determined to be above the threshold, determine that the portion of the first input is phonetically similar to the second input;

upon a condition when the measure of similarity is determined not to be above the threshold, determine that the portion of the first input is not phonetically similar to the second input;

based on determining that the portion of the first input is phonetically similar to the second input, determine an alternative query input by altering the first input based on the second input;

based on determining that the portion of the first input is not phonetically similar to the second input, determine the alternative query input by supplementing the first input with the second input;

select a subset of content items from the set of content items based on comparing the alternative query input and the metadata associated with the subset of content items; and

present the subset of content items to the user.

10. The system of claim 9,

wherein the determination that the portion of the first input has the measure of importance that exceeds the threshold value includes computer-readable instructions causing the system to be configured to identify one or more phrase boundaries in incremental input of the first input, and

wherein the identification of the one or more phrase boundaries is based at least in part on at least one of (a) an identified disfluency from the user in the first input, (b) grammar rules applied to the first input, (c) the importance measure of the portion of the first input, (d) at least one previous conversational interaction with the user, and (e) a user preference signature, and

wherein the user preference signature describes preferences of the user for at least one of (i) particular content items and (ii) particular metadata associated with the content items, wherein the portion of the first input is identified based on the user preference signature,

wherein the disfluency includes at least one of a pause in speech input, an auditory time filler in speech input, and a pause in typing input.

11. The system of claim 9, wherein the selection of the subset of content items is based further upon a disfluency identified in the first input, and based further upon prior conversational interactions that are determined to be related to the first input and the second input.

12. The system of claim 9, wherein the computer-readable instructions causing the system to be configured to provide the feedback includes at least one of:

computer-readable instructions causing the system to be configured to request clarification on the identified portion of the first input,

computer-readable instructions causing the system to be configured to suggest a completion of the received first input, and

computer-readable instructions causing the system to be configured to repeat the portion of the first input to the user, so as to notify the user that the portion of the first input is potentially incorrectly recognized.

13. The system of claim 12, wherein the request for clarification on the identified portion of the first input is based at least in part on a determination that a disfluency occurs after the user has provided the portion of the first input.

14. The system of claim 12, wherein the suggestion of the completion of the received first input is based at least in part on a determination that the disfluency occurs before the user would be expected to provide the portion of the first input.

15. The system of claim 9, wherein the feedback provided to the user is chosen based on at least one of:

a duration of an identified disfluency in the first input,

a measure of confidence in correct speech-to-text recognition of the portion of the first input,

a count of ambiguities detected in the first input,

a count of error corrections needed to identify the portion of the first input,

a count of nodes in a graph data structure, wherein the count of the nodes in the graph data structure measures a path between a first node representing an item of interest from a previous conversational interaction and a second node representing the portion of the first input, and

a measure of relatedness of the portion of the first input to previous conversational interactions with the user.

16. The system of claim 9, wherein the computer-readable instructions causing the system to be configured to present the subset of content items includes computer-readable instructions causing the system to be configured to present the subset of content items before receiving completed input from the user, upon a determination of a strong recognition match for the first input and upon a determination that a responsiveness measure of the selected subset of content items would be above a threshold.

19. The system of claim 9, wherein the computer readable instructions cause the computer system, when determining the measure of similarity between the portion of the first input and the portion of the second input, to determine the measure of similarity between the portion of the first input and the portion of the second input by detecting similar characters in the portion of the first input and the portion of the second input.

20. The system of claim 9, wherein the computer readable instructions cause the computer system, when determining the measure of similarity between the portion of the first input and the portion of the second input, to determine the measure of similarity between the portion of the first input and the portion of the second input by detecting similar phonetic variations in the portion of the first input and the portion of the second input.