Hands-free list-reading by intelligent automated assistant

• US 10,679,605 B2
• Filed: 06/08/2013
• Issued: 06/09/2020
• Est. Priority Date: 01/18/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Gruber, Thomas R., Saddler, Harry J., Napolitano, Lia T., Schubert, Emily Clark, Sumner, Brian Conrad
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): Chavez, Rodrigo A
• Application Number:
  US13/913,423
• Publication Number:
  US 20130275138A1
• Time in Patent Office: 2,558 Days
• Field of Search: 704258, 704260, 704270, 704275
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems

Abstract

Systems and methods for providing hands-free reading of content comprising: identifying a plurality of data items for presentation to a user, the plurality of data items associated with a domain-specific item type and sorted according to a particular order; based on the domain-specific item type, generating a speech-based overview of the plurality of data items; for each of the plurality of data items, generating a respective speech-based, item-specific paraphrase for the data item based on respective content of the data item; and providing, to a user through the speech-enabled dialogue interface, the speech-based overview, followed by the respective speech-based, item-specific paraphrases for at least a subset of the plurality of data items in the particular order.

Claims

1. A method for providing information through a speech-enabled dialogue interface, comprising:

identifying a plurality of data items for presentation to a user, the plurality of data items associated with a domain-specific item type and sorted according to a particular order;

based on the domain-specific item type, generating a speech-based overview of the plurality of data items;

for each of the plurality of data items, generating a respective speech-based, item-specific paraphrase for the data item based on respective content of the data item; and

providing, to a user through the speech-enabled dialogue interface, the speech-based overview, followed by the respective speech-based, item-specific paraphrases for at least a subset of the plurality of data items in the particular order.

2. The method of claim 1, further comprising:

while providing the respective speech-based, item-specific paraphrases, inserting a pause between each pair of adjacent speech-based, item-specific paraphrases; and

entering a listening mode to capture user input during the pause.

3. The method of claim 1, further comprising:

while providing the respective speech-based, item-specific paraphrases in a sequential order, advancing a context marker to a current data item for which the respective speech-based, item-specific paraphrase is being provided to the user.

4. The method of claim 1, further comprising:

receiving user input selecting one of the plurality of data items without specifying any action applicable to the domain-specific item type; and

in response to receiving the user input, providing a speech-based prompt to the user, the speech-based prompt offering one or more action choices applicable to the selected data item.

5. The method of claim 1, further comprising:

for at least one of the plurality of data items, determining a respective size of an unbounded portion of the data item;

in accordance with predetermined criteria, performing one of;

(1) providing a speech-based output reading an entirety of the unbounded portion to the user; and

(2) chunking the unbounded portion of the data item into multiple discrete sections, providing a speech-based output reading a particular discrete section of the multiple discrete sections to the user, and prompting user input regarding whether to read the remaining discrete sections of the multiple discrete sections.

6. The method of claim 1, further comprising:

receiving a user input requesting presentation of the plurality of data items;

processing the user input to determine whether the user has explicitly requested reading of the plurality of data items;

upon determination that the user has explicitly requested reading of the plurality of data items, automatically providing the speech-based, item specific paraphrases following the provision of the speech-based overview without further user request; and

upon determination that the user has not explicitly requested reading of the plurality of data items, prompting a user confirmation before providing the respective speech-based, item-specific paraphrases to the user.

7. The method of claim 1, further comprising:

receiving a user input while providing the speech-based overview and item-specific paraphrases to the user;

processing the speech input to determine whether the speech input relates to the plurality of data items; and

upon determination that the speech input does not relate to the plurality of data items;

suspending output generation related to the plurality of data items, and

providing to the user an output that is responsive to the speech input and unrelated to the plurality of data items.

8. The method of claim 1, wherein the domain-specific item type is reminders and the plurality of data items are a plurality of reminders for a particular time range.

9. The method of claim 8, further comprising:

detecting a trigger event for presenting a listing of reminders to the user; and

in response to the user input, identifying the plurality of reminders to be presented to the user in accordance with one or more relevance criteria, the one or more relevance criteria based on one or more of a current date, a current time, a current location, an action performed by the user or a device associated with the user, an action to be performed by the user or a device associated with the user, and a reminder category specified by the user.

10. The method of claim 9, wherein identifying the plurality of reminders further comprises:

applying additional relevance criteria to select the plurality of reminders until a count of the plurality of reminders no longer exceeds a predetermined threshold number.

11. The method of claim 1, wherein the overview includes the number of data items.

12. The method of claim 1, wherein at least one data item includes a data structure that includes at least one data field containing content;

and

wherein generating a respective speech-based, item-specific paraphrase for the data item based on respective content of the data item includes extracting and restructuring content from at least one data field.

13. A non-transitory computer-readable medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:

identifying a plurality of data items for presentation to a user, the plurality of data items associated with a domain-specific item type and sorted according to a particular order;

based on the domain-specific item type, generating a speech-based overview of the plurality of data items;

for each of the plurality of data items, generating a respective speech-based, item-specific paraphrase for the data item based on respective content of the data item; and

providing, to a user through the speech-enabled dialogue interface, the speech-based overview, followed by the respective speech-based, item-specific paraphrases for at least a subset of the plurality of data items in the particular order.

14. The computer-readable medium of claim 13, wherein the operations further comprise:

while providing the respective speech-based, item-specific paraphrases, inserting a pause between each pair of adjacent speech-based, item-specific paraphrases; and

entering a listening mode to capture user input during the pause.

15. The computer-readable medium of claim 13, wherein the operations further comprise:

while providing the respective speech-based, item-specific paraphrases in a sequential order, advancing a context marker to a current data item for which the respective speech-based, item-specific paraphrase is being provided to the user.

16. The computer-readable medium of claim 13, wherein the operations further comprise:

receiving user input selecting one of the plurality of data items without specifying any action applicable to the domain-specific item type; and

in response to receiving the user input, providing a speech-based prompt to the user, the speech-based prompt offering one or more action choices applicable to the selected data item.

17. The computer-readable medium of claim 13, wherein the operations further comprise:

for at least one of the plurality of data items, determining a respective size of an unbounded portion of the data item;

in accordance with predetermined criteria, performing one of;

(1) providing a speech-based output reading an entirety of the unbounded portion to the user; and

(2) chunking the unbounded portion of the data item into multiple discrete sections, providing a speech-based output reading a particular discrete section of the multiple discrete sections to the user, and prompting user input regarding whether to read the remaining discrete sections of the multiple discrete sections.

18. The computer-readable medium of claim 13, wherein the operations further comprise:

receiving a user input requesting presentation of the plurality of data items;

processing the user input to determine whether the user has explicitly requested reading of the plurality of data items;

upon determination that the user has explicitly requested reading of the plurality of data items, automatically providing the speech-based, item specific paraphrases following the provision of the speech-based overview without further user request; and

upon determination that the user has not explicitly requested reading of the plurality of data items, prompting a user confirmation before providing the respective speech-based, item-specific paraphrases to the user.

19. The computer-readable medium of claim 13, wherein the operations further comprise:

receiving a user input while providing the speech-based overview and item-specific paraphrases to the user;

processing the speech input to determine whether the speech input relates to the plurality of data items; and

upon determination that the speech input does not relate to the plurality of data items;

suspending output generation related to the plurality of data items, and

providing to the user an output that is responsive to the speech input and unrelated to the plurality of data items.

20. The computer-readable medium of claim 13, wherein the domain-specific item type is reminders and the plurality of data items are a plurality of reminders for a particular time range.

21. The computer-readable medium of claim 20, wherein the operations further comprise:

detecting a trigger event for presenting a listing of reminders to the user; and

in response to the user input, identifying the plurality of reminders to be presented to the user in accordance with one or more relevance criteria, the one or more relevance criteria based on one or more of a current date, a current time, a current location, an action performed by the user or a device associated with the user, an action to be performed by the user or a device associated with the user, and a reminder category specified by the user.

22. The computer-readable medium of claim 21, wherein identifying the plurality of reminders further comprises:

applying additional relevance criteria to select the plurality of reminders until a count of the plurality of reminders no longer exceeds a predetermined threshold number.

24. The system of claim 19, wherein the operations further comprise:

while providing the respective speech-based, item-specific paraphrases, inserting a pause between each pair of adjacent speech-based, item-specific paraphrases; and

entering a listening mode to capture user input during the pause.

25. The system of claim 19, wherein the operations further comprise:

while providing the respective speech-based, item-specific paraphrases in a sequential order, advancing a context marker to a current data item for which the respective speech-based, item-specific paraphrase is being provided to the user.

26. The system of claim 19, wherein the operations further comprise:

receiving user input selecting one of the plurality of data items without specifying any action applicable to the domain-specific item type; and

in response to receiving the user input, providing a speech-based prompt to the user, the speech-based prompt offering one or more action choices applicable to the selected data item.

27. The system of claim 19, wherein the operations further comprise:

for at least one of the plurality of data items, determining a respective size of an unbounded portion of the data item;

in accordance with predetermined criteria, performing one of;

(1) providing a speech-based output reading an entirety of the unbounded portion to the user; and

(2) chunking the unbounded portion of the data item into multiple discrete sections, providing a speech-based output reading a particular discrete section of the multiple discrete sections to the user, and prompting user input regarding whether to read the remaining discrete sections of the multiple discrete sections.

28. The system of claim 19, wherein the operations further comprise:

receiving a user input requesting presentation of the plurality of data items;

processing the user input to determine whether the user has explicitly requested reading of the plurality of data items;

upon determination that the user has explicitly requested reading of the plurality of data items, automatically providing the speech-based, item specific paraphrases following the provision of the speech-based overview without further user request; and

upon determination that the user has not explicitly requested reading of the plurality of data items, prompting a user confirmation before providing the respective speech-based, item-specific paraphrases to the user.

29. The system of claim 19, wherein the operations further comprise:

receiving a user input while providing the speech-based overview and item-specific paraphrases to the user;

processing the speech input to determine whether the speech input relates to the plurality of data items; and

upon determination that the speech input does not relate to the plurality of data items;

suspending output generation related to the plurality of data items, and

providing to the user an output that is responsive to the speech input and unrelated to the plurality of data items.

30. The system of claim 19, wherein the domain-specific item type is reminders and the plurality of data items are a plurality of reminders for a particular time range.

31. The system of claim 30, wherein the operations further comprise:

detecting a trigger event for presenting a listing of reminders to the user; and

in response to the user input, identifying the plurality of reminders to be presented to the user in accordance with one or more relevance criteria, the one or more relevance criteria based on one or more of a current date, a current time, a current location, an action performed by the user or a device associated with the user, an action to be performed by the user or a device associated with the user, and a reminder category specified by the user.

32. The system of claim 31, wherein identifying the plurality of reminders further comprises:

applying additional relevance criteria to select the plurality of reminders until a count of the plurality of reminders no longer exceeds a predetermined threshold number.

23. A system, comprising:

one or more processors; and

memory having instructions stored thereon, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising;

identifying a plurality of data items for presentation to a user, the plurality of data items associated with a domain-specific item type and sorted according to a particular order;

based on the domain-specific item type, generating a speech-based overview of the plurality of data items;

for each of the plurality of data items, generating a respective speech-based, item-specific paraphrase for the data item based on respective content of the data item; and

providing, to a user through the speech-enabled dialogue interface, the speech-based overview, followed by the respective speech-based, item-specific paraphrases for at least a subset of the plurality of data items in the particular order.