Intelligent automated assistant

• US 10,741,185 B2
• Filed: 03/13/2019
• Issued: 08/11/2020
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas R., Cheyer, Adam J., Keen, Daniel
• Primary Examiner(s): Joseph, Tonya
• Application Number:
  US16/352,410
• Publication Number:
  US 20190214024A1
• Time in Patent Office: 517 Days
• Field of Search: None
• CPC Class Codes: G06F 16/2457 with adaptation to user needs
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06Q 10/02 Reservations, e.g. for tickets, services or events
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

The intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, and employs additional functionally powered by external services with which the system can interact.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive a first user input;

after receiving the first user input, receive a second input;

in response to receiving the second user input, define a dialogue session including a previous entertainment event search, wherein the dialog session is defined by a predetermined duration based on the first user input and the second user input;

receive a speech input from a user through a dialogue interface of the digital assistant, wherein the speech input is related to an entertainment event search;

determine context information associated with the speech input, wherein the context information includes previously obtained search results related to the previous entertainment event search;

process the speech input to determine one or more entertainment search criteria, wherein the processing comprises at least one of supplementing and disambiguating the speech input with the previously obtained search results to determine at least one of the one or more entertainment search criteria; and

present a plurality of search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the one or more entertainment search criteria.

2. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

receive a reservation request for a first search result of the plurality of search results from the user through the dialogue interface of the digital assistant; and

in response to receiving the reservation request, execute a reservation procedure to purchase one or more tickets for a respective entertainment event corresponding to the first search result.

3. The computer readable storage medium of claim 1, wherein the context information associated with the speech input includes at least one prior interaction between the user and the digital assistant through the dialogue interface.

4. The computer readable storage medium of claim 1, wherein the context information includes information stored in one or more applications including a calendar application, an e-mail application, an instant messaging application, and an address book application.

5. The computer readable storage medium of claim 1, wherein the context information includes one or more of a current location, a current time, and current or forecasted weather information.

6. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

concurrently present one or more user interface elements in the dialogue interface along with the plurality of search results, wherein selection of at least one of the user interface elements causes downloading and playing of a media file associated with a selected search result.

7. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

receive, through the dialogue interface, a follow-up speech input from the user requesting additional information;

disambiguate at least part of the follow-up speech input to generate a search query for retrieving the requested additional information, wherein the disambiguating is based at least in part on one or more properties related to respective entertainment events corresponding to the plurality of search results; and

present information retrieved based on the search query to the user through the dialogue interface of the digital assistant.

8. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

during a dialogue session in which the speech input was received;

maintain a dialogue context based on user inputs received through the dialogue interface and information provided to the user through the dialogue interface;

receive one or more additional speech inputs from the user through the dialogue interface;

process each of the one or more additional speech inputs based on the dialogue context to determine a user intent expressed in the additional speech input, the processing including at least one of supplementing and disambiguating the additional speech input based on the dialogue context; and

based on the processing, provide additional information in response to each of the one or more additional speech inputs.

9. The computer readable storage medium of claim 8, wherein the instructions further cause the device to:

during the dialogue session in which the speech input was received;

receive a reservation request through the dialogue interface of the digital assistant after presentation of the plurality of search results has been replaced with other information in response to subsequent user inputs;

determine a target entertainment event of the reservation request based on the dialogue context, wherein the target entertainment event corresponds to a first search result of the plurality of search results; and

in response to receiving the reservation request, execute a reservation procedure to purchase one or more tickets for the determined target entertainment event.

10. The computer readable storage medium of claim 1, wherein the context information includes information extracted from one or more speech inputs previously received through the dialogue interface.

11. The computer readable storage medium of claim 1, wherein the context information includes information extracted from one or more non-verbal inputs previously received through the dialogue interface.

12. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

provide an audio output including a paraphrase of a request included in the speech input.

13. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

provide an audio output including at least one of the plurality of search results.

14. The computer readable storage medium of claim 1, wherein the instructions further cause the device to:

determine whether at least one search result of the plurality of search results satisfy the one or more entertainment search criteria; and

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

modify at least one search criteria of the one or more entertainment search criteria.

15. The computer readable storage medium of claim 14, wherein modifying at least one search criteria of the one or more entertainment search criteria includes expanding a location search area.

16. The computer readable storage medium of claim 14, wherein the instructions further cause the device to:

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

present a plurality of second search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the modified one or more entertainment search criteria.

17. The computer readable storage medium of claim 16, wherein the plurality of second search results includes an indication that at least one search result of the plurality of search results did not satisfy the one or more entertainment search criteria.

18. The computer readable storage medium of claim 16, wherein the plurality of second search results includes an indication that the one or more entertainment search criteria was modified.

19. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a first user input;

after receiving the first user input, receiving a second input;

in response to receiving the second user input, defining a dialogue session including a previous entertainment event search, wherein the dialog session is defined by a predetermined duration based on the first user input and the second user input;

receiving a speech input from a user through a dialogue interface of the digital assistant, wherein the speech input is related to an entertainment event search;

determining context information associated with the speech input, wherein the context information includes previously obtained search results related to the previous entertainment event search;

processing the speech input to determine one or more entertainment search criteria, wherein the processing comprises at leak one of supplementing and disambiguating the speech input with the previously obtained search results to determine at least one of the one or more entertainment search criteria; and

presenting a plurality of search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the one or more entertainment search criteria.

21. The device of claim 19, wherein the one or more programs further include instructions for:

receiving a reservation request for a first search result of the plurality of search results from the user through the dialogue interface of the digital assistant; and

in response to receiving the reservation request, executing a reservation procedure to purchase one or more tickets for a respective entertainment event corresponding to the first search result.

22. The device of claim 19, wherein the context information associated with the speech input includes at least one prior interaction between the user and the digital assistant through the dialogue interface.

23. The device of claim 19, wherein the context information includes information stored in one or more applications including a calendar application, an e-mail application, an instant messaging application, and an address book application.

24. The device of claim 19, wherein the context information includes one or more of a current location, a current time, and current or forecasted weather information.

25. The device of claim 19, wherein the one or more programs further include instructions for:

concurrently presenting one or more user interface elements in the dialogue interface along with the plurality of search results, wherein selection of at least one of the user interface elements causes downloading and playing of a media file associated with a selected search result.

26. The device of claim 19, wherein the one or more programs further include instructions for:

receiving, through the dialogue interface, a follow-up speech input from the user requesting additional information;

disambiguating at least part of the follow-up speech input to generate a search query for retrieving the requested additional information, wherein the disambiguating is based at least in part on one or more properties related to respective entertainment events corresponding to the plurality of search results; and

presenting information retrieved based on the search query to the user through the dialogue interface of the digital assistant.

27. The device of claim 19, wherein the one or more programs further include instructions for:

during a dialogue session in which the speech input was received;

maintaining a dialogue context based on user inputs received through the dialogue interface and information provided to the user through the dialogue interface;

receiving one or more additional speech inputs from the user through the dialogue interface;

processing each of the one or more additional speech inputs based on the dialogue context to determine a user intent expressed in the additional speech input, the processing including at least one of supplementing and disambiguating the additional speech input based on the dialogue context; and

based on the processing, providing additional information in response to each of the one or more additional speech inputs.

28. The device of claim 27, wherein the one or more programs further include instructions for:

during the dialogue session in which the speech input was received;

receiving a reservation request through the dialogue interface of the digital assistant after presentation of the plurality of search results has been replaced with other information in response to subsequent user inputs;

determining a target entertainment event of the reservation request based on the dialogue context, wherein the target entertainment event corresponds to a first search result of the plurality of search results; and

in response to receiving the reservation request, executing a reservation procedure to purchase one or more tickets for the determined target entertainment event.

29. The device of claim 19, wherein the context information includes information extracted from one or more speech inputs previously received through the dialogue interface.

30. The device of claim 19, wherein the context information includes information extracted from one or more non-verbal inputs previously received through the dialogue interface.

31. The device of claim 19, wherein the one or more programs further include instructions for:

providing an audio output including a paraphrase of a request included in the speech input.

32. The device of claim 19, wherein the one or more programs further include instructions for:

providing an audio output including at least one of the plurality of search results.

33. The device of claim 19, wherein the one or more programs further include instructions for:

determining whether at least one search result of the plurality of search results satisfy the one or more entertainment search criteria; and

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

modifying at least one search criteria of the one or more entertainment search criteria.

34. The device of claim 33, wherein modifying at least one search criteria of the one or more entertainment search criteria includes expanding a location search area.

35. The device of claim 33, wherein the one or more programs further include instructions for:

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

presenting a plurality of second search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the modified one or more entertainment search criteria.

36. The device of claim 35, wherein the plurality of second search results includes an indication that at least one search result of the plurality of search results did not satisfy the one or more entertainment search criteria.

37. The device of claim 35, wherein the plurality of second search results includes an indication that the one or more entertainment search criteria was modified.

20. A method, comprising:

at an electronic device with one or more processors and memory;

receiving a first user input;

after receiving the first user input, receiving a second input;

in response to receiving the second user input defining a dialogue session including a previous entertainment event search, wherein the dialog session is defined by a predetermined duration based on the first user input and the second user input;

receiving a speech input from a user through a dialogue interface of the digital assistant, wherein the speech input is related to an entertainment event search;

determining context information associated with the speech input, wherein the context information includes previously obtained search results related to the previous entertainment event search;

processing the speech input to determine one or more entertainment search criteria, wherein the processing comprises at least one of supplementing and disambiguating the speech input with the previously obtained search results to determine at least one of the one or more entertainment search criteria; and

presenting a plurality of search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the one or more entertainment search criteria.

38. The method of claim 20, further comprising:

receiving a reservation request for a first search result of the plurality of search results from the user through the dialogue interface of the digital assistant; and

in response to receiving the reservation request, executing a reservation procedure to purchase one or more tickets for a respective entertainment event corresponding to the first search result.

39. The method of claim 20, wherein the context information associated with the speech input includes at least one prior interaction between the user and the digital assistant through the dialogue interface.

40. The method of claim 20, wherein the context information includes information stored in one or more applications including a calendar application, an e-mail application, an instant messaging application, and an address book application.

41. The method of claim 20, wherein the context information includes one or more of a current location, a current time, and current or forecasted weather information.

42. The method of claim 20, further comprising:

concurrently presenting one or more user interface elements in the dialogue interface along with the plurality of search results, wherein selection of at least one of the user interface elements causes downloading and playing of a media file associated with a selected search result.

43. The method of claim 20, further comprising:

receiving, through the dialogue interface, a follow-up speech input from the user requesting additional information;

disambiguating at least part of the follow-up speech input to generate a search query for retrieving the requested additional information, wherein the disambiguating is based at least in part on one or more properties related to respective entertainment events corresponding to the plurality of search results; and

presenting information retrieved based on the search query to the user through the dialogue interface of the digital assistant.

44. The method of claim 20, further comprising:

during a dialogue session in which the speech input was received;

maintaining a dialogue context based on user inputs received through the dialogue interface and information provided to the user through the dialogue interface;

receiving one or more additional speech inputs from the user through the dialogue interface;

processing each of the one or more additional speech inputs based on the dialogue context to determine a user intent expressed in the additional speech input, the processing including at least one of supplementing and disambiguating the additional speech input based on the dialogue context; and

based on the processing, providing additional information in response to each of the one or more additional speech inputs.

45. The method of claim 44, further comprising:

during the dialogue session in which the speech input was received;

receiving a reservation request through the dialogue interface of the digital assistant after presentation of the plurality of search results has been replaced with other information in response to subsequent user inputs;

determining a target entertainment event of the reservation request based on the dialogue context, wherein the target entertainment event corresponds to a first search result of the plurality of search results; and

in response to receiving the reservation request, executing a reservation procedure to purchase one or more tickets for the determined target entertainment event.

46. The method of claim 20, wherein the context information includes information extracted from one or more speech inputs previously received through the dialogue interface.

47. The method of claim 20, wherein the context information includes information extracted from one or more non-verbal inputs previously received through the dialogue interface.

48. The method of claim 20, further comprising:

providing an audio output including a paraphrase of a request included in the speech input.

49. The method of claim 20, further comprising:

providing an audio output including at least one of the plurality of search results.

50. The method of claim 20, further comprising:

determining whether at least one search result of the plurality of search results satisfy the one or more entertainment search criteria; and

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

modifying at least one search criteria of the one or more entertainment search criteria.

51. The method of claim 50, wherein modifying at least one search criteria of the one or more entertainment search criteria includes expanding a location search area.

52. The method of claim 50, further comprising:

in accordance with a determination that at least one search result of the plurality of search results does not satisfy the one or more entertainment search criteria;

presenting a plurality of second search results to the user through the dialogue interface, each search result corresponding to a respective entertainment event identified in accordance with the modified one or more entertainment search criteria.

53. The method of claim 52, wherein the plurality of second search results includes an indication that at least one search result of the plurality of search results did not satisfy the one or more entertainment search criteria.

54. The method of claim 52, wherein the plurality of second search results includes an indication that the one or more entertainment search criteria was modified.