Intelligent automated assistant for delivering content from user experiences

• US 11,145,294 B2
• Filed: 08/07/2018
• Issued: 10/12/2021
• Est. Priority Date: 05/07/2018
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Vescovi, Marcos Regis, Circlaeys, Eric M. G., Warren, Richard, Bernstein, Jeffrey Traer, Krenn, Matthaeus
• Primary Examiner(s): Washburn, Daniel C
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US16/057,396
• Publication Number:
  US 20190341027A1
• Time in Patent Office: 1,162 Days
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications

Abstract

Systems and processes for operating an intelligent automated assistant are provided. In one example process, a speech input is received from a user. In response to determining that the speech input corresponds to a user intent of obtaining information associated with a user experience of the user, one or more parameters referencing a user experience of the user are identified. Metadata associated with the referenced user experience is obtained from an experiential data structure. Based on the metadata, one or more media items associated with the referenced are retrieved based on the metadata. The one or more media items associated with the referenced user experience are output together.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of an electronic device, cause the electronic device to:

receive, from a user, speech input;

determine whether the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

in response to a determination that the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

identify, from the speech input, one or more parameters referencing a user experience of the user;

obtain, from an experiential data structure, metadata associated with the referenced user experience;

retrieve, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience; and

output together the one or more media items associated with the referenced user experience;

determine at least one pattern associated with the user; and

in accordance with a determination that the at least one pattern associated with the user satisfies one or more predetermined criteria, update the experiential data structure.

2. The non-transitory computer-readable storage medium of claim 1, wherein determining whether the speech input corresponds to a user intent of obtaining information associated with a user experience further comprises:

determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience.

3. The non-transitory computer-readable storage medium of claim 2, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes one or more nouns related to images and one or more nouns related to an experience.

4. The non-transitory computer-readable storage medium of claim 2, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes at least one semantic term for an experience, at least one verb for a past activity, and at least one verb for a search.

5. The non-transitory computer-readable storage medium of claim 1, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter type corresponding to an event, an activity, a geographical feature, a person, a place, a time, and a location.

6. The non-transitory computer-readable storage medium of claim 1, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter sub-type corresponding to at least one parameter type.

7. The non-transitory computer-readable storage medium of claim 1, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one descriptor corresponding to at least one parameter type.

8. The non-transitory computer-readable storage medium of claim 1, wherein retrieving, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience further comprises:

retrieving media items associated with a user profile.

9. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

in accordance with a determination that at least one predetermined criterion is satisfied, retrieve media items related to the referenced user experience from a remote source.

10. The non-transitory computer-readable storage medium of claim 9, wherein the at least one predetermined criterion includes that a number of retrieved media items associated with the referenced user experience is less than a predetermined threshold.

11. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based on at least one pattern recognition process.

12. The non-transitory computer-readable storage medium of claim 1, wherein updating the experiential data structure further comprises:

determining whether a user experience associated with the at least one pattern was previously generated;

in accordance with a determination that a user experience associated with the at least one pattern was previously generated, updating the previously generated user experience; and

in accordance with a determination that a user experience associated with the at least one pattern was not previously generated, generating a user experience associated with the at least one pattern.

13. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based at least on user data obtained from a secondary device.

14. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based at least on user motion data.

15. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based at least on user biometric data.

16. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based at least on facial recognition data.

17. The non-transitory computer-readable storage medium of claim 1, wherein the instructions further cause the one or more processors to:

generate the experiential data structure based at least on user contact data.

18. The non-transitory computer-readable storage medium of claim 1, wherein outputting together the one or more media items associated with the referenced user experience further comprises:

generating, using the metadata, information associated with the referenced user experience; and

outputting together the one or more media items associated with the referenced user experience and the generated information associated with the referenced user experience.

19. An electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving, from a user, speech input;

determining whether the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

in response to a determination that the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

identifying, from the speech input, one or more parameters referencing a user experience of the user;

obtaining, from an experiential data structure, metadata associated with the referenced user experience;

retrieving, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience; and

outputting together the one or more media items associated with the referenced user experience;

determining at least one pattern associated with the user; and

in accordance with a determination that the at least one pattern associated with the user satisfies one or more predetermined criteria, updating the experiential data structure.

20. The device of claim 19, wherein determining whether the speech input corresponds to a user intent of obtaining information associated with a user experience further comprises:

determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience.

21. The device of claim 20, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes one or more nouns related to images and one or more nouns related to an experience.

22. The device of claim 20, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes at least one semantic term for an experience, at least one verb for a past activity, and at least one verb for a search.

23. The device of claim 19, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter type corresponding to an event, an activity, a geographical feature, a person, a place, a time, and a location.

24. The device of claim 19, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter sub-type corresponding to at least one parameter type.

25. The device of claim 19, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one descriptor corresponding to at least one parameter type.

26. The device of claim 19, wherein retrieving, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience further comprises:

retrieving media items associated with a user profile.

27. The device of claim 19, the one or more programs including instructions for:

in accordance with a determination that at least one predetermined criterion is satisfied, retrieve media items related to the referenced user experience from a remote source.

28. The device of claim 27, wherein the at least one predetermined criterion includes that a number of retrieved media items associated with the referenced user experience is less than a predetermined threshold.

29. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based on at least one pattern recognition process.

30. The device of claim 19, wherein updating the experiential data structure further comprises:

determining whether a user experience associated with the at least one pattern was previously generated;

in accordance with a determination that a user experience associated with the at least one pattern was previously generated, updating the previously generated user experience; and

in accordance with a determination that a user experience associated with the at least one pattern was not previously generated, generating a user experience associated with the at least one pattern.

31. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based at least on user data obtained from a secondary device.

32. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based at least on user motion data.

33. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based at least on user biometric data.

34. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based at least on facial recognition data.

35. The device of claim 19, the one or more programs including instructions for:

generating the experiential data structure based at least on user contact data.

36. The device of claim 19, wherein outputting together the one or more media items associated with the referenced user experience further comprises:

generating, using the metadata, information associated with the referenced user experience; and

outputting together the one or more media items associated with the referenced user experience and the generated information associated with the referenced user experience.

37. A method, comprising:

at an electronic device with one or more processors and memory;

receiving, from a user, speech input;

determining whether the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

in response to a determination that the speech input corresponds to a user intent of obtaining information associated with a user experience of the user;

identifying, from the speech input, one or more parameters referencing a user experience of the user;

obtaining, from an experiential data structure, metadata associated with the referenced user experience;

retrieving, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience; and

outputting together the one or more media items associated with the referenced user experience;

determining at least one pattern associated with the user; and

in accordance with a determination that the at least one pattern associated with the user satisfies one or more predetermined criteria, updating the experiential data structure.

38. The method of claim 37, wherein determining whether the speech input corresponds to a user intent of obtaining information associated with a user experience further comprises:

determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience.

39. The method of claim 38, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes one or more nouns related to images and one or more nouns related to an experience.

40. The method of claim 38, wherein determining whether the speech input corresponds to a user intent of obtaining media associated with the user experience further comprises:

determining whether the speech input includes at least one semantic term for an experience, at least one verb for a past activity, and at least one verb for a search.

41. The method of claim 37, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter type corresponding to an event, an activity, a geographical feature, a person, a place, a time, and a location.

42. The method of claim 37, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one parameter sub-type corresponding to at least one parameter type.

43. The method of claim 37, wherein identifying, from the speech input, one or more parameters referencing a user experience further comprises:

determining whether the speech input includes at least one descriptor corresponding to at least one parameter type.

44. The method of claim 37, wherein retrieving, based on the metadata associated with the referenced user experience, one or more media items associated with the referenced user experience further comprises:

retrieving media items associated with a user profile.

45. The method of claim 37, comprising:

in accordance with a determination that at least one predetermined criterion is satisfied, retrieve media items related to the referenced user experience from a remote source.

46. The method of claim 45, wherein the at least one predetermined criterion includes that a number of retrieved media items associated with the referenced user experience is less than a predetermined threshold.

47. The method of claim 37, comprising:

generating the experiential data structure based on at least one pattern recognition process.

48. The method of claim 37, wherein updating the experiential data structure further comprises:

determining whether a user experience associated with the at least one pattern was previously generated;

in accordance with a determination that a user experience associated with the at least one pattern was previously generated, updating the previously generated user experience; and

in accordance with a determination that a user experience associated with the at least one pattern was not previously generated, generating a user experience associated with the at least one pattern.

49. The method of claim 37, comprising:

generating the experiential data structure based at least on user data obtained from a secondary device.

50. The method of claim 37, the comprising:

generating the experiential data structure based at least on user motion data.

51. The method of claim 37, comprising:

generating the experiential data structure based at least on user biometric data.

52. The method of claim 37, comprising:

generating the experiential data structure based at least on facial recognition data.

53. The method of claim 37, comprising:

generating the experiential data structure based at least on user contact data.

54. The method of claim 37, wherein outputting together the one or more media items associated with the referenced user experience further comprises:

generating, using the metadata, information associated with the referenced user experience; and

outputting together the one or more media items associated with the referenced user experience and the generated information associated with the referenced user experience.