Data driven natural language event detection and classification

• US 10,521,466 B2
• Filed: 09/19/2016
• Issued: 12/31/2019
• Est. Priority Date: 06/11/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bellegarda, Jerome R., Martel, Mathieu Jean, Carlhian, Alexandre, Vinel, Antoine, Dolfing, Jannes G., Rekik, Sabrine, Berget, Markus, Arras, Hafid
• Primary Examiner(s): Conyers, Dawaune A
• Application Number:
  US15/269,721
• Publication Number:
  US 20170357716A1
• Time in Patent Office: 1,198 Days
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/358 Browsing; Visualisation therefor
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L

Abstract

Systems and processes for operating a digital assistant are provided. In accordance with one or more examples, a method includes, at a user device with one or more processors and memory, receiving unstructured natural language information from at least one user. The method also includes, in response to receiving the unstructured natural language information, determining whether event information is present in the unstructured natural language information. The method further includes, in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information. The method further includes, in accordance with a determination that an agreement on an event is present, determining an event type of the event and providing an event description based on the event type.

Claims

1. A method, comprising:

at an electronic device including at least one processor;

receiving unstructured natural language information from at least one user;

in response to receiving the unstructured natural language information, determining whether event information is present in the unstructured natural language information, wherein determining whether event information is present in the unstructured language information comprises;

determining one or more polarities associated with the unstructured natural language information using a neural network;

determining whether the one or more polarities include at least one of a proposal, a rejection, or an acceptance;

in accordance with a determination that the one or more polarities include at least one of a proposal, a rejection, or an acceptance, determining a probability associated with the at least one of the proposal, the rejection, or the acceptance;

determining whether the probability satisfies a first probability threshold; and

in accordance with a determination that the probability satisfies the first probability threshold, determining that event information is present in the unstructured natural language information;

in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information;

in accordance with a determination that an agreement on an event is present, determining an event type of the event;

providing an event description based on the event type; and

generating a calendar entry based on the event description.

2. The method of claim 1, wherein the unstructured natural language information comprises one or more messages.

3. The method of claim 1, wherein determining one or more polarities associated with the unstructured natural language information using the neural network comprises:

generating an input layer based on the unstructured natural language information, wherein the input layer comprises a plurality of input units and a plurality of preceding first-context units;

generating a hidden layer based on the input layer, wherein the hidden layer comprises a plurality of current first-context units; and

generating an output layer based on the hidden layer, wherein the output layer includes one or more output label units representing the one or more polarities of at least a portion of the unstructured natural language information.

4. The method of claim 3, wherein generating the input layer based on the unstructured natural language information comprises:

obtaining a word sequence based on the unstructured natural language information;

obtaining a plurality of tokens based on the unstructured natural language information; and

generating the input layer using at least one of the word sequence and the plurality of tokens.

5. The method of claim 3, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a first weight matrix;

weighting a preceding first-context unit using a second weight matrix; and

determining a current first-context unit based on the weighting of the current input unit and the weighting of the preceding first-context unit.

6. The method of claim 3, wherein the hidden layer further comprises a plurality of second-context units.

7. The method of claim 6, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a third weight matrix;

weighting a following second-context unit using a fourth weight matrix; and

determining a current second-context unit based on the weighting of the current input unit and the weighting of the following second-context unit.

8. The method of claim 3, wherein the output layer comprises a plurality of output label units, each output label unit has an 1-of-K encoding, wherein the K represents a number of polarities of a pre-determined polarity set.

9. The method of claim 8, wherein the pre-determined polarity set comprises a proposal, a rejection, an acceptance, and a no-event.

10. The method of claim 3, wherein generating the output layer based on the hidden layer comprises:

weighting a current first-context unit using a fifth weight matrix; and

determining a current output label unit based on the weighting of the current first-context unit.

11. The method of claim 3, wherein generating an output layer based on the hidden layer further comprises:

obtaining a current state based on a current first-context unit and a current second-context unit;

weighting the current state using a sixth weight matrix; and

determining a current output label unit based on the weighting of the current state.

12. The method of claim 1, in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information comprises:

determining whether the one or more polarities include an acceptance;

in accordance with a determination that the one or more polarities include an acceptance, determining a probability associated with the acceptance;

determining whether the probability satisfies a second probability threshold; and

in accordance with a determination that probability satisfies the second probability threshold, determining that an agreement on an event is present.

13. The method of claim 1, wherein determining the event type is based on a hierarchical classification using a natural language event ontology.

14. The method of claim 13, wherein the natural language event ontology comprises one or more levels, each level including one or more event type nodes.

15. The method of claim 14, wherein at least one event type node is associated with a priority rule, the priority rule indicating a relative priority of the at least one event type node with respect to another event type node.

16. The method of claim 1, wherein determining the event type of the event comprises:

determining a first-level event type associated with the event; and

determining the event type of the event based on the first-level event type.

17. The method of claim 16, wherein determining the first-level event type comprises determining a first-level event type output of an output layer, wherein the first-level event type output has an 1-of-L encoding, wherein the L represents a number of first-level event type nodes of the natural language event ontology.

18. The method of claim 17, wherein determining the first-level event type output comprises:

weighting a trailing first-context unit of a hidden layer; and

determining the first-level event type output based on the weighting of the trailing first-context unit.

19. The method of claim 17, wherein determining the first-level event type output comprises:

concatenating a trailing first-context unit and a leading second-context unit;

weighting the concatenation of the trailing first-context unit and the leading second-context unit; and

determining the first-level event type output based on the weighting of the concatenation of the trailing first-context unit and the leading second-context unit.

20. The method of claim 17, wherein determining the first-level event type output comprises applying an activation function.

21. The method of claim 16, wherein determining the event type of the event based on the first-level event type comprises:

determining, for each second-level event type node, a number of correlations between the second-level event type node and the unstructured natural language information, wherein each second-level event type node is a child node of a first-level event type node representing the first-level event type; and

determining a second-level event type based on the number of correlations of each second-level event type node.

22. The method of claim 21, wherein determining the second-level event type comprises:

determining, based on the number of correlations of each second-level event type node, a maximum number of correlations; and

determining the second-level event type based on the maximum number of correlations.

23. The method of claim 22, further comprising:

determining whether at least two second-level event type nodes correspond to the maximum number of correlations; and

in accordance with the determination that at least two second-level event type nodes correspond to the maximum number of correlations, determining the second-level event type based on one or more priority rules.

24. The method of claim 1, wherein providing the event description based on the event type of the event comprises providing at least one of:

an event title, a starting time of the event, an ending time of the event, and a duration of the event, wherein the event title includes a subject of a corresponding event type node.

25. The method of claim 1, further comprising:

displaying the calendar entry; and

providing one or more affordances, wherein the one or more affordances enable receiving one or more user inputs with respect to the calendar entry.

26. The method of claim 1, wherein the event description comprises event information present in the unstructured language information.

27. The method of claim 1, wherein the received unstructured natural language information comprises one or more messages from at least two users and wherein the agreement on an event is an agreement between the at least two users determined from the one or more messages.

28. A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors of an electronic device, the one or more programs including instructions for:

receiving unstructured natural language information from at least one user;

in response to receiving the unstructured natural language information, determining whether event information is present in the unstructured natural language information, wherein determining whether event information is present in the unstructured language information comprises;

determining one or more polarities associated with the unstructured natural language information using a neural network ;

determining whether the one or more polarities include at least one of a proposal, a rejection, or an acceptance;

in accordance with a determination that the one or more polarities include at least one of a proposal, a rejection, or an acceptance, determining a probability associated with the at least one of the proposal, the rejection, or the acceptance;

determining whether the probability satisfies a first probability threshold; and

in accordance with a determination that the probability satisfies the first probability threshold, determining that event information is present in the unstructured natural language information;

in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information;

in accordance with a determination that an agreement on an event is present, determining an event type of the event;

providing an event description based on the event type; and

generating a calendar entry based on the event description.

29. The non-transitory computer-readable storage medium of claim 28, wherein determining one or more polarities associated with the unstructured natural language information using the neural network comprises:

generating an input layer based on the unstructured natural language information, wherein the input layer comprises a plurality of input units and a plurality of preceding first-context units;

generating a hidden layer based on the input layer, wherein the hidden layer comprises a plurality of current first-context units; and

generating an output layer based on the hidden layer, wherein the output layer includes one or more output label units representing the one or more polarities of at least a portion of the unstructured natural language information.

30. The non-transitory computer-readable storage medium of claim 29, wherein generating the input layer based on the unstructured natural language information comprises:

obtaining a word sequence based on the unstructured natural language information;

obtaining a plurality of tokens based on the unstructured natural language information; and

generating the input layer using at least one of the word sequence and the plurality of tokens.

31. The non-transitory computer-readable storage medium of claim 29, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a first weight matrix;

weighting a preceding first-context unit using a second weight matrix; and

determining a current first-context unit based on the weighting of the current input unit and the weighting of the preceding first-context unit.

32. The non-transitory computer-readable storage medium of claim 29, wherein the hidden layer further comprises a plurality of second-context units.

33. The non-transitory computer-readable storage medium of claim 32, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a third weight matrix;

weighting a following second-context unit using a fourth weight matrix; and

determining a current second-context unit based on the weighting of the current input unit and the weighting of the following second-context unit.

34. The non-transitory computer-readable storage medium of claim 29, wherein generating the output layer based on the hidden layer comprises:

weighting a current first-context unit using a fifth weight matrix; and

determining a current output label unit based on the weighting of the current first-context unit.

35. The non-transitory computer-readable storage medium of claim 29, wherein generating an output layer based on the hidden layer further comprises:

obtaining a current state based on a current first-context unit and a current second-context unit;

weighting the current state using a sixth weight matrix; and

determining a current output label unit based on the weighting of the current state.

36. The non-transitory computer-readable storage medium of claim 28, in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information comprises:

determining whether the one or more polarities include an acceptance;

in accordance with a determination that the one or more polarities include an acceptance, determining a probability associated with the acceptance;

determining whether the probability satisfies a second probability threshold; and

in accordance with a determination that probability satisfies the second probability threshold, determining that an agreement on an event is present.

37. The non-transitory computer-readable storage medium of claim 28, wherein determining the event type is based on a hierarchical classification using a natural language event ontology.

38. The non-transitory computer-readable storage medium of claim 28, wherein determining the event type of the event comprises:

determining a first-level event type associated with the event; and

determining the event type of the event based on the first-level event type.

39. The non-transitory computer-readable storage medium of claim 38, wherein determining the event type of the event based on the first-level event type comprises:

determining, for each second-level event type node, a number of correlations between the second-level event type node and the unstructured natural language information, wherein each second-level event type node is a child node of a first-level event type node representing the first-level event type; and

determining a second-level event type based on the number of correlations of each second-level event type node.

40. The non-transitory computer-readable storage medium of claim 28, wherein providing the event description based on the event type of the event comprises providing at least one of:

an event title, a starting time of the event, an ending time of the event, and a duration of the event, wherein the event title includes a subject of a corresponding event type node.

41. The non-transitory computer-readable storage medium of claim 28, wherein the one or more programs further include instructions for:

displaying the calendar entry; and

providing one or more affordances, wherein the one or more affordances enable receiving one or more user inputs with respect to the calendar entry.

42. A system, comprising:

one or more processors;

memory; and

one or more programs stored in memory and for execution by the one or more processors, the one or more programs including instructions for;

receiving unstructured natural language information from at least one user;

in response to receiving the unstructured natural language information, determining whether event information is present in the unstructured natural language information, wherein determining whether event information is present in the unstructured language information comprises;

determining one or more polarities associated with the unstructured natural language information using a neural network;

determining whether the one or more polarities include at least one of a proposal, a rejection, or an acceptance;

in accordance with a determination that the one or more polarities include at least one of a proposal, a rejection, or an acceptance, determining a probability associated with the at least one of the proposal, the rejection, or the acceptance;

determining whether the probability satisfies a first probability threshold; and

in accordance with a determination that the probability satisfies the first probability threshold, determining that event information is present in the unstructured natural language information;

in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information;

in accordance with a determination that an agreement on an event is present, determining an event type of the event;

providing an event description based on the event type; and

generating a calendar entry based on the event description.

43. The system of claim 42, wherein determining one or more polarities associated with the unstructured natural language information using the neural network comprises:

generating an input layer based on the unstructured natural language information, wherein the input layer comprises a plurality of input units and a plurality of preceding first-context units;

generating a hidden layer based on the input layer, wherein the hidden layer comprises a plurality of current first-context units; and

generating an output layer based on the hidden layer, wherein the output layer includes one or more output label units representing the one or more polarities of at least a portion of the unstructured natural language information.

44. The system of claim 43, wherein generating the input layer based on the unstructured natural language information comprises:

obtaining a word sequence based on the unstructured natural language information;

obtaining a plurality of tokens based on the unstructured natural language information; and

generating the input layer using at least one of the word sequence and the plurality of tokens.

45. The system of claim 43, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a first weight matrix;

weighting a preceding first-context unit using a second weight matrix; and

determining a current first-context unit based on the weighting of the current input unit and the weighting of the preceding first-context unit.

46. The system of claim 43, wherein the hidden layer further comprises a plurality of second-context units.

47. The system of claim 46, wherein generating the hidden layer based on the input layer comprises:

weighting a current input unit using a third weight matrix;

weighting a following second-context unit using a fourth weight matrix; and

determining a current second-context unit based on the weighting of the current input unit and the weighting of the following second-context unit.

48. The system of claim 43, wherein generating the output layer based on the hidden layer comprises:

weighting a current first-context unit using a fifth weight matrix; and

determining a current output label unit based on the weighting of the current first-context unit.

49. The system of claim 43, wherein generating an output layer based on the hidden layer further comprises:

obtaining a current state based on a current first-context unit and a current second-context unit;

weighting the current state using a sixth weight matrix; and

determining a current output label unit based on the weighting of the current state.

50. The system of claim 42, in accordance with a determination that event information is present within the unstructured natural language information, determining whether an agreement on an event is present in the unstructured natural language information comprises:

determining whether the one or more polarities include an acceptance;

in accordance with a determination that the one or more polarities include an acceptance, determining a probability associated with the acceptance;

determining whether the probability satisfies a second probability threshold; and

in accordance with a determination that probability satisfies the second probability threshold, determining that an agreement on an event is present.

51. The system of claim 42, wherein determining the event type is based on a hierarchical classification using a natural language event ontology.

52. The system of claim 42, wherein determining the event type of the event comprises:

determining a first-level event type associated with the event; and

determining the event type of the event based on the first-level event type.

53. The system of claim 52, wherein determining the event type of the event based on the first-level event type comprises:

determining, for each second-level event type node, a number of correlations between the second-level event type node and the unstructured natural language information, wherein each second-level event type node is a child node of a first-level event type node representing the first-level event type; and

determining a second-level event type based on the number of correlations of each second-level event type node.

54. The system of claim 42, wherein providing the event description based on the event type of the event comprises providing at least one of:

an event title, a starting time of the event, an ending time of the event, and a duration of the event, wherein the event title includes a subject of a corresponding event type node.

55. The system of claim 42, wherein the one or more programs further include instructions for:

displaying the calendar entry; and

providing one or more affordances, wherein the one or more affordances enable receiving one or more user inputs with respect to the calendar entry.