SYSTEMS AND METHODS FOR ENHANCING USER EXPERIENCE BY COMMUNICATING TRANSIENT ERRORS

• US 20190080685A1
• Filed: 09/08/2017
• Published: 03/14/2019
• Est. Priority Date: 09/08/2017
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Johnson, JR., Charles Melvin
• Granted Patent: US 11,004,444 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 30/0256 User search
• CPC Class Codes: G10L 15/18 using natural language modelling
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: H04M 3/4936 Speech interaction details speech recognition per se G10L15/00

Abstract

This patent application is directed to interaction between a voice-controlled device and a language processing system that processes user requests. The requests can be requests for information or they can be requests to control a device, such as “Turn on the bedroom light.” The disclosure includes various embodiments for trying to resolve system errors in a manner that provides an improved customer experience. The improvements described include the capability of the overall system to recognize that an error has occurred while attempting to fulfill a request that was submitted as part of a recognized spoken utterance. Instead of simply doing nothing, or of playing a message to the customer to try again later, the system would now let the user know it was still processing the request while making another attempt at fulfilling the request. This requires the system to retain the entire utterance until it is determined that the request has been fulfilled or that the request cannot be fulfilled, so that the customer does not have to repeat the request a second (or third) time. The feedback to the customer can be audio, visual, or a combination. For example, the Echo could play an audio message “please stand by, I'"'"'m still working on your request” while at the same time, the colored ring could turn green to indicate all is well and processing is continuing.

Claims

1. A method, comprising:

receiving, from an electronic device, first audio data representing a first utterance that includes a request;

determining a user account associated with the electronic device;

generating, using automatic speech recognition (ASR) processing, text data representing the first audio data;

generating, using natural language understanding (NLU) processing, intent data representing the text data, the intent data corresponding to an intent associated with the request;

determining a first skill to process the intent data;

sending a first instruction to the first skill to process the intent data;

determining that a first error occurred in the first skill processing the intent data by receiving a failure data in response to sending the first instruction;

generating, in response to determining that the first error occurred, output text data representing language indicating the error occurred and that a second attempt to respond to the request is being made;

generating, using text-to-speech (TTS) processing, output audio data from the output text data;

sending the output audio data to the electronic device for presentation; and

determining a second skill to process the intent data.

3. The method of claim 1, further comprising:

sending a second instruction to the first skill to process the intent data;

determining that an error occurred in the first skill processing the intent data;

generating, in response to determining that the second error occurred, additional output text data representing language indicating another error occurred and that a third attempt to respond to the request is being made;

generating, using TTS processing, additional output audio data from the additional output text data;

sending the additional output audio data to the electronic device for presentation;

selecting the second skill to process the intent data; and

sending a third instruction to the second skill to process the intent data.

4. The method of claim 3, the first skill being previously assigned as a first choice skill for processing the intent, the method further comprising:

determining that the second skill successfully processed the intent data; and

assigning the second skill as the first-choice skill for processing the intent.

5. The method of claim 1, further comprising:

determining that the second skill is substantially different from the first skill;

generating text data representing an inquiry for permission to use the second skill to process the intent data;

generating, using TTS processing, output audio data from the output text data; and

sending the output audio data to the electronic device for presentation.

2. The method of claim 2, further comprising:

assessing the error to determine the first skill is capable of processing the intent data within a predetermined temporal interval;

determining that the error prevents the first skill from processing the intent data within the predetermined temporal interval; and

sending a second instruction to the second skill to process the intent data.

6. A method, comprising:

receiving, from an electronic device, first audio data representing a first utterance that includes a request;

determining, from the first audio data, first intent data representing the request;

determining, using the first intent data, a first functionality;

initiating, using the first functionality, a first attempt to process the first intent data;

determining that a first error occurred during the first attempt;

generating first failure data indicating that the first attempt failed to process the first intent data and that a second attempt to process the first intent data is being made;

sending the first failure data to the electronic device for presentation;

determining, using the first intent data, a second functionality; and

initiating a second attempt to process the first intent data.

7. The method of claim 6, wherein the first intent data corresponds to an intent, the first functionality being previously assigned as the first-choice functionality for processing the intent, further comprising:

assigning the second functionality as the first-choice functionality for processing the intent.

8. The method of claim 6, wherein assigning comprises:

determining, based on historical data, that previous attempts to process the intent using the first functionality have resulted in a predetermined frequency of errors;

sending instructions to the electronic device to output audio data representing an inquiry for permission to assign the second functionality as the first-choice functionality for processing the intent;

receiving second audio data representing an utterance; and

assigning, in response to receiving the second audio data, the second functionality as the first-choice functionality for processing the intent.

9. The method of claim 6, further comprising:

sending instructions to the electronic device to output audio data representing an inquiry for permission to use the second functionality to process the first intent data;

receiving, from the electronic device, second audio data representing a second utterance; and

initiating the second attempt being in response to receiving the second audio data.

10. The method of claim 9, the second audio data representing an utterance that includes a second request, further comprising:

determining, based on the second audio data, second intent data;

determining that the second functionality is capable of processing the second intent data; and

initiating the second attempt comprising sending instructions to the second functionality to process the second intent data.

11. The method of claim 6, further comprising:

determining that a second error occurred during the second attempt;

generating second failure data indicating that the first and second attempts to process the first intent data have failed, the second failure data further representing an inquiry for permission to initiate repeated attempts to process the first intent data until the first intent data is successfully processed;

receiving second audio data representing a second utterance; and

initiating, based on the second audio data, a third attempt to process the first intent data.

12. The method of claim 6, further comprising:

assessing, prior to initiating the second attempt, the error to determine additional attempts to process the intent data can be made within a predetermined temporal interval; and

determining, based on assessing the error, that additional attempts to process the intent data can be made within the predetermined temporal interval.

13. The method of claim 6, wherein the first intent data corresponds to an intent, the second attempt comprising sending instructions to the first functionality to process the intent, the method further comprising:

determining that an error occurred during the second attempt;

generating second failure data indicating that the second attempt failed to process the first intent data and that a third attempt to process the first intent data is being made;

sending the second failure data to the electronic device for presentation;

initiating a third attempt to fulfill the request, the third attempt comprising sending instructions to the second functionality to process the first intent data.

14. The method of claim 13, further comprising:

determining that the first intent data represents a request to use visual information;

determining a user account associated with the electronic device;

determining that a display screen is associated with the electronic device;

determining a user preference associated with the user account; and

displaying the first notification via the display screen, the first notification comprising, based on the user preference, at least one of;

an image, a series of images, and a video.

15. The method of claim 14, the at least one image, series of images, and video being randomly selected from a database located on a user device that is separate from the electronic device and associated with the user account.

16. The method of claim 6, further comprising:

generating a record, the record including a timestamp, the first audio data, identification information associated with the electronic device; and

information regarding the first and second attempts to process the first intent data; and

sending the record to a database.

17. A system, comprising:

communications circuitry that receives, from an electronic device, first audio data representing an utterance that includes a request; and

at least one processor operable to;

determine, from the first audio data, first intent data representing the request;

determine, from using the first intent data, a first functionality;

initiate, using the first functionality, a first attempt to fulfill the request;

determine that an error occurred in the first attempt;

generate first failure data indicating that the first attempt failed to process the first intent data and that a second attempt to process the first intent data is being made;

send the first failure data to the electronic device for presentation;

determine, using the first intent data, a second functionality; and

initiate a second attempt to process the first intent data.

18. The system of claim 17, wherein the first intent data corresponds to an intent, the first functionality being previously assigned as the first-choice functionality for processing the intent, the at least one processor being further operable to:

assign the second functionality as the first-choice functionality for processing the intent.

19. The system of claim 18, further comprising a display screen, the at least one processor being further operable to:

determine that the request includes a request to use visual information;

determine a user account associated with the electronic device;

determine that the electronic device is associated with the display screen; and

determine a user preference associated with the user account; and

display the first failure data via the display screen, the first failure data representing, based on the user preference, at least one of;

an image, a series of images, and a video.

20. The system of claim 19, further comprising:

a user device associated with the user account, the user device comprising;

memory; and

data representing at least one of;

an image, series of images, and video stored in memory, wherein the at least one image, series of images, and video are stored in the memory of the user device.