Systems and methods for collaborative note-taking

• US 20050171926A1
• Filed: 02/02/2004
• Published: 08/04/2005
• Est. Priority Date: 02/02/2004
• Status: Active Grant

• Current Assignee: Fuji Xerox, Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Denoue, Laurent, Thione, Giovanni Lorenzo, Van Den Berg, Martin Henk
• Granted Patent: US 7,542,971 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/166 Editing, e.g. inserting or deleting
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Techniques are provided for determining collaborative notes and automatically recognizing speech, handwriting and other type of information. Domain and optional actor/speaker information associated with the support information is determined. An initial automatic speech recognition model is determined based on the domain and/or actor information. The domain and/or actor/speaker language model is used to recognize text in the speech information associated with the support information. Presentation support information such as slides, speaker notes and the like are determined. The semantic overlap between the support information and the salient non-function words in the recognized text and collaborative user feedback information are used to determine relevancy scores for the recognized text. Grammaticality, well formedness, self referential integrity and other features are used to determine correctness scores. Suggested collaborative notes are displayed in the user interface based on the salient non-function words. User actions in the user interface determine feedback signals. Recognition models such as automatic speech recognition, handwriting recognition are determined based on the feedback signals and the correctness and relevance scores.

Claims

1. A method for recognizing information from an information source comprising the steps of:

determining portions of information from a first ambiguous information source;

determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

determining output information based on at least one of the determined recognition models.

3. The method of claim 1, wherein the recognition model recognizes at least one of:

speech and handwriting information.

4. The method of claim 1, wherein the recognition model recognizes features associated with at least one of:

audio information, video information, and tactile information.

5. The method of claim 1, wherein the recognition model recognizes attributes associated with at least one of:

a song, a book and a video.

6. The method of claim 5, wherein the attributes are at least one of:

a title, a genre, an author.

7. The method of claim 1, wherein the context information is at least one of support information and collaborative user feedback information.

8. The method of claim 1, wherein the recognition model is a composite recognition mode.

9. The method of claim 8, wherein the composite recognition model is comprised of at least one of an actor based model, a domain based model and a genre based model.

10. The method of claim 9, wherein the actor based model is a speaker based recognition model.

11. The method of claim 1, wherein the output information is a suggested note.

12. The method of claim 7, wherein the support information is based on at least one of:

audio, visual, tactile information.

13. The method of claim 12, wherein the audio information is collaborative user generated audio.

14. The method of claim 12, wherein the visual information is at least one of video information, video capture of user gestures, textual information.

15. The method of claim 7, wherein the collaborative user feedback is based on user actions.

16. The method of claim 15, wherein the user actions operate on at least one of:

suggested collaborative notes, user notes, user corrections, user pauses.

17. The method of claim 15, wherein a user action is ignoring a suggested note.

18. The method of claim 15, wherein the collaborative user feedback is based on at least one of:

numbers of user actions, identity of users taking action, weighting based on which users take action.

19. The method of claim 12, wherein the tactile information is displayed using a dynamically refreshable tactile display.

20. The method of claim 12, wherein the tactile information is displayed using a dynamically refreshable tactile display.

21. The method of claim 1, wherein the at least one recognition model is dynamically determined.

22. The method of claim 18, wherein the at least one composite recognition model is determined dynamically.

23. The method of claim 7, wherein the collaborative feedback information is determined based on at least one of:

intensity of user response, number of users responding, type of responses, time to respond, identity of user responding.

116. The method of claim 10, further comprising the step of determining matching the verbalization of acronyms in the first information source with the tokenized representation of the acronym in the second information source.

117. The method of claim 10, further comprising the step of determining matching the verbalization of acronyms in the second information source with the tokenized representation of the acronym in the first information source.

2. A method for recognizing information from an information source comprising the steps of:

determining portions of information from a first information source;

determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

determining output information based on at least one of the determined recognition models.

24. A method for displaying collaborative user notes comprising the steps of:

determining portions of a first information channel;

dynamically determining domain information based on the determined portions;

determining context information temporally associated with the determined portions;

determining a suggested note based on collaborative user feedback information and at least one of;

the determined portions of the first information channel and the temporally related context information.

25. The method of claim 24, in which portions of the first information channel contain ambiguities.

26. The method of claim 24, in which a recognition model provides alternate salient tokens for recognized portions of the first information channel.

27. The method of claim 24, wherein determining domain information comprises determining salient tokens based on at least one of:

the first information channel, the context information channel and the at least one recognition model.

28. The method of claim 25, further comprising the steps of:

determining at least one of a relevance score and a correctness score between alternate portions of each ambiguous portion of the first information channel and associated portions of the context information channel; and

determining an ordering of alternate portions based on at least one of the relevance score and the correctness score.

29. The method of claim 25, further comprising the steps of:

determining at least one of a relevance score and a correctness score between alternate hypotheses for each portion of the first information channel and associated portions of the context information channel; and

determining an ordering of alternate hypotheses based on at least one of the relevance score and the correctness score.

30. The method of claim 28, wherein determining the relevance score comprises determining a relatedness metric between portions of the first information channel and the context information channel.

31. The method of claim 30, wherein the relatedness metric is at least one of a literal similarity metric and conceptual similarity metric.

32. The method of claim 31, wherein the conceptual similarity is based on a lexicon.

33. The method of claim 32, wherein the lexicon is the WordNet lexicon.

34. The method of claim 29, wherein the correctness score is determined based on a correctness metric based on a well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis.

35. The method of claim 29, wherein the correctness score is based on collaborative user feedback.

36. The method of claim 29, wherein the correctness score is based on an intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined.

37. The method of claim 29, wherein the correctness score is based on phonetic overlap.

38. The method of claim 29, wherein the correctness score is based on a predicted flow through the context information.

39. The method of claim 29, wherein the correctness score is based on the presence of acronyms and associated expansions in the first information and the context information.

40. The method of claim 29, wherein the context information is comprised of user notes and the correctness information is based on at least one of:

semantic and lexical distance from the user notes.

41. A method of ranking tokens recognized from an information source comprising ranking the salient tokens based on correctness based on at least one of:

well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis, collaborative user feedback, an intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined, phonetic overlap, and a predicted flow through the context information.

42. A method of displaying suggested meeting notes comprising the steps of:

determining recognized speech information;

determining contextually related support information;

determining salient tokens in the recognized speech information based on a similarity metric and wherein the similarity metric indicates the similarity between recognized speech information and the contextually related support information;

determining output information based on the salient tokens, the speech information and the contextually related support information; and

determining output display attributes based on the similarity metric.

43. The method of claim 42, wherein the output display attribute is zooming in portions of at least one of:

the speech information, the context information, a reproduction or transformation of the context information and the salient tokens.

44. The method of claim 42, wherein the output display attribute is at least one of:

highlighting, fonts, color, italics, arrows, pointers, and blinking.

45. The method of claim 42, further comprising the step of ranking alternate salient tokens for display based on at least one of a relevancy metric and a correctness metric.

46. The method of claim 42, further comprising the step of displaying the ranked alternate salient tokens based on at least one of time, relevance score, and correctness score.

47. The method of claim 46, wherein the ranked alternate salient tokens are displayed along a temporal axis.

48. The method of claim 47, wherein the ranked alternate salient tokens are clustered based on time intervals.

49. The method of claim 48, wherein the time intervals are dynamically determined.

50. The method of claim 49, wherein the time intervals are dynamically determined based on user actions.

51. The method of claim 42, further comprising the step of dynamically determining a selection area associated with each ranked alternate salient 5 token and based on at least one of the relevancy score and the correctness score.

52. A system for recognizing information from an information source, the system comprising:

a memory;

an input/output circuit for retrieving portions of information from a first information source and portions of context information from a second information source and wherein the portions of information from the first information source are temporally associated with the portions of context information from the second information source;

a processor that determines at least one recognition model based on the portions of information from the first information source and the temporally associated portion of context information from the second information source and which determines output information based on at least one recognition model.

53. A system for recognizing information from an information source, the system comprising:

a memory;

an input/output circuit for retrieving portions of information from a first information source and portions of context information from a second information source and wherein the portions of information from the first information source are temporally associated with portions of context information from the second information source useful in disambiguating the portions of information from the first information source;

a processor that determines at least one recognition model based on the portions of information from the first information source and the temporally associated portion of context information from the second information source and which determines output information based on at least one recognition model.

54. A system for recognizing information from an information source, the system comprising:

a memory;

an input/output circuit for retrieving portions of information from a first information source and portions of context information from a second information source and wherein the portions of information from the first information source are temporally associated with portions of context information from the second information source;

a processor that determines at least one composite recognition model based on the portions of information from the first information source and the temporally associated portion of context information from the second information source, and which determines output information based on at least one recognition model.

55. The system of claim 54, wherein the recognition model recognizes at least one of speech and handwriting information.

56. The system of claim 54, wherein the recognition model recognizes features associated with at least one of:

audio information, video information, and tactile information.

57. The system of claim 54, wherein the recognition model recognizes attributes associated with at least one of:

a song, a book and a video.

58. The system of claim 57, wherein the attributes are at least one of:

a title, a genre, an author.

59. The system of claim 54, wherein the context information is at least one of support information and collaborative user feedback information.

60. The system of claim 54, wherein the recognition model is a composite recognition mode.

61. The system of claim 60, wherein the composite recognition model is comprised of at least one of an actor based model, a domain based model and a genre based model.

62. The system of claim 61, wherein the actor based model is a speaker based recognition model.

63. The system of claim 54, wherein the output information is a suggested note.

64. The system of claim 59, wherein the support information is based on at least one of:

audio, visual, tactile information.

65. The system of claim 64, wherein the audio information is collaborative user generated audio.

66. The system of claim 64, wherein the visual information is at least one of video information, video capture of user gestures, textual information.

67. The system of claim 59, wherein the collaborative user feedback is based on user actions.

68. The system of claim 67, wherein the user actions operate on at least one of:

suggested collaborative notes, user notes, user corrections, user pauses.

69. The system of claim 67, wherein a user action is ignoring a suggested note.

70. The system of claim 67, wherein the collaborative user feedback is based on at least one of:

numbers of user actions, identity of users taking action, weighting based on which users take action.

71. The system of claim 64, wherein the tactile information is displayed using a dynamically refreshable tactile display.

72. The system of claim 64, wherein the tactile information is displayed using a dynamically refreshable tactile display.

73. The system of claim 54, wherein the at least one recognition model is dynamically determined.

74. The system of claim 70, wherein the at least one composite recognition model is determined dynamically.

75. The system of claim 59, wherein the collaborative feedback information is determined based on at least one of:

intensity of user response, number of users responding, type of responses, time to respond, identity of user responding.

118. The system of claim 62, further comprising the step of determining matching the verbalization of acronyms in the first information source with the tokenized representation of the acronym in the second information source.

119. The system of claim 62, further comprising the step of determining matching the verbalization of acronyms in the second information source with the tokenized representation of the acronym in the first information source.

76. A system for displaying collaborative user notes comprising:

a memory;

an input/output circuit for retrieving portions of information from a first information channel and portions of context information from a second information channel and wherein the portions of information from the first information channel are temporally associated with portions of context information from the second information channel;

a processor that dynamically determines domain information associated with the portions of information from the first information channel based on the portions of information from the first information channel and the temporally associated portions of context information from the second information channel and which determines a suggested note based on a collaborative user feedback signal ad at least one of;

the determined portions of the first information channel and the temporally associated portions of context information from the second information channel.

77. The system of claim 76, in which portions of the first information channel contain ambiguities.

78. The system of claim 76, in which a recognition model provides alternate salient tokens for recognized portions of the first information channel.

79. The system of claim 76, wherein determining domain information comprises determining salient tokens based on at least one of:

the first information channel, the context information channel and the at least one recognition model.

80. The system of claim 77, further comprising the steps of:

determining at least one of a relevance score and a correctness score between alternate portions of each ambiguous portion of the first information channel and associated portions of the context information channel; and

determining an ordering of alternate portions based on at least one of the relevance score and the correctness score.

81. The system of claim 77, further comprising the steps of:

determining at least one of a relevance score and a correctness score between alternate hypotheses for each portion of the first information channel and associated portions of the context information channel; and

determining an ordering of alternate hypotheses based on at least one of the relevance score and the correctness score.

82. The system of claim 80, wherein determining the relevance score comprises determining a relatedness metric between portions of the first information channel and the context information channel.

83. The system of claim 82, wherein the relatedness metric is at least one of a literal similarity metric and conceptual similarity metric.

84. The system of claim 83, wherein the conceptual similarity is based on a lexicon.

85. The system of claim 84, wherein the lexicon is the WordNet®

lexicon.

86. The system of claim 80, wherein the correctness score is determined based on a correctness metric based on a well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis.

87. The system of claim 80, wherein the correctness score is based on collaborative user feedback.

88. The system of claim 80, wherein the correctness score is based on an the intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined.

89. The system of claim 80, wherein the correctness score is based on phonetic overlap.

90. The system of claim 80, wherein the correctness score is based on a predicted flow through the context information.

91. The system of claim 80, wherein the correctness score is based on the presence of acronyms and associated expansions in the first information and the context information.

92. The system of claim 80, wherein the context information is comprised of user notes and the correctness information is based on at least one of:

semantic and lexical distance from the user notes.

93. A system of ranking tokens recognized from an information source comprising:

a memory;

an input/output circuit; and

a processor that ranks salient tokens received by the input/output circuit based on correctness based on at least one of;

well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis, collaborative user feedback, an intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined, phonetic overlap, and a predicted flow through the context information.

94. A system of displaying suggested meeting notes comprising:

a memory;

an input/output circuit for retrieving recognized speech information and contextually related support information;

a processor that determines salient tokens in the recognized speech information based on a similarity metric and which indicates the similarity between the recognized speech information and the contextually related support information;

determines output information based on the salient tokens, the speech information and the contextually related support information; and

that determines output display attributes based on the similarity metric.

95. The system of claim 94, wherein the output display attribute is zooming in portions of at least one of:

the speech information, the context information, a reproduction or transformation of the context information and the salient tokens.

96. The system of claim 94, wherein the output display attribute is at least one of:

highlighting, fonts, color, italics, arrows, pointers, and blinking.

97. The system of claim 94, further comprising a ranking of alternate salient tokens for display based on at least one of a relevancy metric and a correctness metric.

98. The system of claim 94, further comprising the displaying of ranked alternate salient tokens based on at least one of time, relevance score, and correctness score.

99. The system of claim 98, wherein the ranked alternate salient tokens are displayed along a temporal axis.

100. The system of claim 99, wherein the ranked alternate salient tokens are clustered based on time intervals.

101. The system of claim 100, wherein the time intervals are dynamically determined.

102. The system of claim 101, wherein the time intervals are dynamically determined based on user actions.

103. The system of claim 101, further comprising dynamically determining a selection area associated with each ranked alternate salient token and based on at least one of the relevancy score and the correctness score.

104. Computer readable storage medium comprising:

computer readable program code embodied on the computer readable storage medium, the computer readable program code usable to program a computer to recognize ambiguous information comprising the steps of;

determining portions of information from a first ambiguous information source;

determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

determining output information based on at least one of the determined recognition models.

105. Computer readable storage medium comprising:

computer readable program code embodied on the computer readable storage medium, the computer readable program code usable to program a computer to recognize ambiguous information comprising the steps of;

determining portions of information from a first information source;

determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

determining output information based on at least one of the determined recognition models.

106. Computer readable storage medium comprising:

computer readable program code embodied on the computer readable storage medium, the computer readable program code usable to program a computer to display collaborative user notes comprising the steps of;

determining portions of a first information channel;

dynamically determining domain information based on the determined portions;

determining context information temporally associated with the determined portions;

determining a suggested note based on collaborative user feedback information and at least one of;

the determined portions of the first information channel and the temporally related context information.

107. Computer readable storage medium comprising:

computer readable program code embodied on the computer readable storage medium, the computer readable program code usable to program a computer to rank tokens recognized from an information source comprising the steps of;

ranking the salient tokens based on correctness based on at least one of;

well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis, collaborative user feedback, an intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined, phonetic overlap, and a predicted flow through the context information.

108. Computer readable storage medium comprising:

computer readable program code embodied on the computer readable storage medium, the computer readable program code usable to program a computer to display suggested meeting notes comprising the steps of;

determining recognized speech information;

determining contextually related support information;

determining salient tokens in the recognized speech information based on a similarity metric and wherein the similarity metric indicates the similarity between recognized speech information and the contextually related support information;

determining output information based on the salient tokens, the speech information and the contextually related support information; and

determining output display attributes based on the similarity metric.

109. A carrier wave encoded to transmit a control program, useable to program a computer to recognize ambiguous information, to a device for executing the program, the control program comprising:

instructions for determining portions of information from a first ambiguous information source;

instructions for determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

instructions for determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

instructions for determining output information based on at least one of the determined recognition models.

110. A carrier wave encoded to transmit a control program, useable to program a computer to recognize information, to a device for executing the program, the control program comprising:

instructions for determining portions of information from a first information source;

instructions for determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

instructions for determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

instructions for determining output information based on at least one of the determined recognition models.

111. A carrier wave encoded to transmit a control program, useable to program a computer to display collaborative user notes, to a device for executing the program, the control program comprising:

instructions for determining portions of a first information channel;

instructions for dynamically determining domain information based on the determined portions;

instructions for determining context information temporally associated with the determined portions;

instructions for determining a suggested note based on collaborative user feedback information and at least one of;

the determined portions of the first information channel and the temporally related context information.

112. A carrier wave encoded to transmit a control program, useable to program a computer to rank tokens recognized from an information source, to a device for executing the program, the control program comprising instructions for ranking the salient tokens based on correctness based on at least one of:

well-formedness, grammaticality, semantics, syntax, discourse structure and discourse analysis, collaborative user feedback, an intrinsic probability that the ambiguous portions of the first information channel and the context information are disjoined, phonetic overlap, and a predicted flow through the context information.

113. A carrier wave encoded to transmit a control program, useable to program a computer to display suggested meeting notes, to a device for executing the program, the control program comprising

instructions for determining recognized speech information;

instructions for determining contextually related support information;

instructions for determining salient tokens in the recognized speech information based on a similarity metric and wherein the similarity metric indicates the similarity between recognized speech information and the contextually related support information;

instructions for determining output information based on the salient tokens, the speech information and the contextually related support information; and

instructions for determining output display attributes based on the similarity metric.

114. A system for recognizing information comprising:

means for determining portions of information from a first ambiguous information source;

means for determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

means for determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

means for determining output information based on at least one of the determined recognition models.

115. A system for recognizing information comprising:

means for determining portions of information from a first information source;

determining portions of context information from a second information source temporally associated with the portions of information from the first information source;

determining at least one recognition model based on the portions of information from the first information source and the temporally associated portions of context information from the second information source;

determining output information based on at least one of the determined recognition models.