PROACTIVE ASSISTANT WITH MEMORY ASSISTANCE

• US 20170091612A1
• Filed: 04/28/2016
• Published: 03/30/2017
• Est. Priority Date: 09/30/2015
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: VESCOVI, Marcos Regis, GUZZONI, Didier R., GRUBER, Thomas R., SKINDER, Jason A.
• Application Number:
  US15/141,716
• Publication Number:
  US 20170091612A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 25/63 for estimating an emotional state

Abstract

A non-transitory computer-readable storage medium stores one or more programs including instructions, which when executed by an electronic device of a user, cause the electronic device to generate at least one experiential data structure accessible to a virtual assistant; modify at least one experiential data structure with one or more annotations associated with the experiential data structure, utilizing the virtual assistant; store at least one experiential data structure; receive a natural-language user request for service from the virtual assistant, and output information responsive to the user request using at least one experiential data structure. The experiential data structure is a data structure that includes an organized set of data associated with at least one of the user and the electronic device at a particular point in time.

Claims

1. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by an electronic device of a user, cause the electronic device to:

generate, in response to a trigger, at least one experiential data structure accessible to a virtual assistant, wherein the experiential data structure comprises an organized set of data associated with at least one of the user and the electronic device at a particular point in time;

store at least one experiential data structure;

modify at least one experiential data structure with one or more annotations associated with the experiential data structure, utilizing the virtual assistant;

receive a natural-language user request for service from the virtual assistant, and

output information responsive to the user request using at least one experiential data structure.

2. The non-transitory computer-readable storage medium of claim 1, wherein the trigger is the passage of a time interval and wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

generate an experiential data structure upon the passage of each time interval.

3. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

modify at least one experiential data structure based on at least one device context.

4. The non-transitory computer-readable storage medium of claim 3, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

detect a change in device context; and

in response to detection of a change in device context, modify at least one experiential data structure based on at least one changed device context.

5. The non-transitory computer-readable storage medium of claim 3, wherein the device context includes a location of the device.

6. The non-transitory computer-readable storage medium of claim 3, wherein the device context includes motion of the device.

7. The non-transitory computer-readable storage medium of claim 3, wherein the device context includes proximity to a second electronic device.

8. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

modify at least one experiential data structure based on at least one user context.

9. The non-transitory computer-readable storage medium of claim 8, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

detect a change in user context; and

in response to detection of a change in user context, modify at least one experiential data structure based on at least one changed user context.

10. The non-transitory computer-readable storage medium of claim 8, wherein the user context includes personal information associated with the user.

11. The non-transitory computer-readable storage medium of claim 8, wherein the user context includes locations associated with the user.

12. The non-transitory computer-readable storage medium of claim 1,wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

wherein the trigger is a user request, receive an express user request to generate at least one experiential data structure; and

in response to receipt of the express user request, generate at least one experiential data structure.

13. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

modify the at least one experiential data structure based on express user input.

14. The non-transitory computer-readable storage medium of claim 13, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

analyze the content of the express user input;

based on the analysis of the content of the express user input, determine whether the user request is ambiguous;

in accordance with a determination that the user request is other than ambiguous, perform the action to modify at least one experiential data structure; and

in accordance with a determination that the user request is ambiguous;

request additional information from the user to disambiguate;

receive the additional information from the user; and

based in part on the additional information from the user, perform the action to modify at least one experiential data structure.

15. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes social information comprising information associated with at least one person other than the user.

16. The non-transitory computer-readable storage medium of claim 15, wherein the social information includes the content of email accessible to the virtual assistant.

17. The non-transitory computer-readable storage medium of claim 15, wherein the social information includes the content of text messages accessible by the virtual assistant.

18. The non-transitory computer-readable storage medium of claim 15, wherein the social information includes the characteristics of calendar events accessible by the virtual assistant.

19. The non-transitory computer-readable storage medium of claim 15, wherein the social information includes contacts accessible by the virtual assistant.

20. The non-transitory computer-readable storage medium of claim 15, wherein the social information includes notes about people accessible by the virtual assistant.

21. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes location information.

22. The non-transitory computer-readable storage medium of claim 21, wherein the location information includes information associated with a period of time during which the electronic device is generally stationary at a location.

23. The non-transitory computer-readable storage medium of claim 21, wherein the location information includes information associated with a period of time during which the electronic device is generally in motion.

24. The non-transitory computer-readable storage medium of claim 21, wherein the location information includes information associated with the frequency with which the electronic device is at a particular location.

25. The non-transitory computer-readable storage medium of claim 21, wherein the location information includes information associated with a user-identified location.

26. The non-transitory computer-readable storage medium of claim 21, wherein the location information includes a location of an object associated with the electronic device.

27. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes media information.

28. The non-transitory computer-readable storage medium of claim 27, wherein the media information includes information associated with a podcast played via the electronic device.

29. The non-transitory computer-readable storage medium of claim 27, wherein the media information includes information associated with music played via the electronic device.

30. The non-transitory computer-readable storage medium of claim 27, wherein the media information includes information associated with video played via the electronic device.

31. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes content information.

32. The non-transitory computer-readable storage medium of claim 31, wherein the content information includes a browser history of the electronic device.

33. The non-transitory computer-readable storage medium of claim 31, wherein the content information includes content received through a browser at the electronic device.

34. The non-transitory computer-readable storage medium of claim 31, wherein the content information includes documents generated by the user with the electronic device.

35. The non-transitory computer-readable storage medium of claim 31, wherein the content information includes a history of application usage at the electronic device.

36. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes photographic information.

37. The non-transitory computer-readable storage medium of claim 1, wherein at least one experiential data structure includes daily activity information.

38. The non-transitory computer-readable storage medium of claim 37, wherein the daily activity information includes reminders accessible to the virtual assistant.

39. The non-transitory computer-readable storage medium of claim 37, wherein the daily activity information includes at least one of diet and exercise information accessible to the virtual assistant.

40. The non-transitory computer-readable storage medium of claim 37, wherein the daily activity information includes user journal information accessible to the virtual assistant.

41. The non-transitory computer-readable storage medium, of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

generate at least one new experiential data structure when at least one of the items of information of the experiential data structure, the device context, and the user context changes.

42. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

receive a user request for service from the virtual assistant associated with at least one stored experiential data structure;

analyze at least one stored experiential data structure based on at least one element of the user request; and

output information responsive to the user request based on the analysis of at least stored one experiential data structure.

43. The non-transitory computer-readable storage medium of claim 42, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

match the user request directly to one or more stored experiential data structures.

44. The non-transitory computer-readable storage medium of claim 42, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

generate at least one additional element based on at least one element of the user request; and

match the generated element to at least one stored experiential data structure.

45. The non-transitory computer-readable storage medium of claim 44, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

generate at least one further additional element, based on the at least one additional element; and

repeat the instruction to generate at least one further additional element, based on the at least one additional element, at least one additional time.

46. The non-transitory computer-readable storage medium of claim 42, wherein the instructions, which when executed by the one or more processors of the electronic device, cause the device to analyze at least one stored experiential data structure based on the user request, further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

analyze statistically a plurality of experiential data structures based on at least one element of the user request.

47. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

analyze the content of the user request;

based on the analysis of the user request, determine whether the user request is ambiguous;

in accordance with a determination that the user request is other than ambiguous, proceed to output information responsive to the user request; and

in accordance with a determination that the user request is ambiguous;

request additional information from the user to disambiguate;

receive the additional information from the user; and

based in part on the additional information from the user, proceed to output information responsive to the user request.

48. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

receive a user request for a recommendation from the virtual assistant;

analyze at least one stored experiential data structure based on the user request; and

output information responsive to the user request based on the analysis of the at least one stored experiential data structure.

49. The non-transitory computer-readable storage medium of claim 48, wherein the instructions to analyze at least one stored experiential data structure based on the user request, further comprise instructions which when executed by the one or more processors of the electronic device, cause the device to:

access, using the virtual assistant, tags associated with anonymized stored experiential data structures of other users; and

analyze, using the virtual assistant, the anonymized stored experiential data structures of other users based on the user request.

50. The non-transitory computer-readable storage medium of claim 48, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

anonymize at least one experiential data structure; and

transmit at least one anonymized experiential data structure from the electronic device.

51. The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

store at least one experiential data structure for a fixed period of time.

52. The non-transitory computer-readable storage medium of claim 51, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

set the fixed period of time independent of the user.

53. The non-transitory computer-readable storage medium of claim 51, wherein the one or more programs further comprise instructions, which when executed by the one or more processors of the electronic device, cause the device to:

receive a period of time selected by the user; and

set the fixed period of time in accordance with the selection received from the user.

54. An electronic device, comprising:

a memory;

a processor coupled to the memory, the processor configured to;

generate, in response to a trigger, at least one experiential data structure accessible to a virtual assistant, wherein the experiential data structure comprises an organized set of data associated with at least one of the user and the electronic device at a particular point in time;

store at least one experiential data structure;

modify at least one experiential data structure with one or more annotations associated with the experiential data structure, utilizing the virtual assistant;

receive a natural-language user request for service from the virtual assistant; and

output information responsive to the user request using at least one experiential data structure.

55. A method of using a virtual assistant, comprising:

at an electronic device configured to transmit and receive data;

generating, in response to a trigger, at least one experiential data structure accessible to a virtual assistant, wherein the experiential data structure comprises an organized set of data associated with at least one of the user and the electronic device at a particular point in time;

storing at least one experiential data structure;

modifying at least one experiential data structure with one or more annotations associated with the experiential data structure, utilizing the virtual assistant;

receiving a natural-language user request for service from the virtual assistant; and

outputting information responsive to the user request using at least one experiential data structure.