Text processing system

• US 20030055655A1
• Filed: 01/28/2002
• Published: 03/20/2003
• Est. Priority Date: 07/17/1999
• Status: Active Grant

• Current Assignee: Buffalo Patents, LLC
• Original Assignee: Optical Research Partners LLC (Intellectual Ventures LLC)
• Inventors: Suominen, Edwin A.
• Granted Patent: US 6,904,405 B2
• Time in Patent Office: Days
• US Class Current: 704/276
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Certain disclosed methods and systems perform multiple different types of message recognition using a shared language model. Message recognition of a first type is performed responsive to a first type of message input (e.g., speech), to provide text data in accordance with both the shared language model and a first model specific to the first type of message recognition (e.g., an acoustic model). Message recognition of a second type is performed responsive to a second type of message input (e.g., handwriting), to provide text data in accordance with both the shared language model and a second model specific to the second type of message recognition (e.g., a model that determines basic units of handwriting conveyed by freehand input). Accuracy of both such message recognizers can be improved by user correction of misrecognition by either one of them. Numerous other methods and systems are also disclosed.

Claims

1. A system responsive to voice input and freehand input to generate and edit text, the system comprising:

(a) a stylus comprising;

(1) a pointer for communicating freehand input; and

(2) a microphone for receiving voice input;

(b) a speech recognizer responsive to voice input to provide text data;

(c) a first tablet interface including a first tablet surface, the first tablet interface being responsive to freehand input to provide indicia of editing commands; and

(d) an editor having access to the text data, responsive to the indicia of editing commands to modify the text data as specified by the editing commands.

2. The system of claim 1 wherein:

(a) the stylus has a proximal end and a distal end;

(b) the pointer is toward the distal end; and

(c) the microphone is toward the proximal end.

3. The system of claim 1 wherein the microphone is responsive to sound impinging on the stylus from the direction of the proximal end with a first sensitivity and from the direction of the distal end with a second sensitivity, the first sensitivity being significantly greater than the second sensitivity.

4. The system of claim 1 further comprising a first display integrated into the first tablet surface.

5. The system of claim 4 further comprising:

(a) a second tablet interface including a second tablet surface; and

(b) a second display integrated into the second tablet surface.

6. The system of claim 5 further comprising:

(a) a communications link interconnecting the displays; and

(b) a command interpreter for controlling the communications link, the command interpreter being responsive to the indicia of editing commands to transfer text data between the displays as specified by the editing commands.

7. The system of claim 1 wherein the speech recognizer is part of a message recognizer that further comprises a handwriting recognizer, the handwriting recognizer being responsive to freehand input to further provide text data.

8. The system of claim 7 wherein:

(c) the speech recognizer generates text data in accordance with an acoustic model and a shared language model; and

(d) the handwriting recognizer generates text data in accordance with a handwriting model and the shared language model; and

(e) the message recognizer is responsive to the indicia of editing commands to adapt at least the shared language model.

9. The system of claim 8 further comprising a character interface responsive to character input communicated via the pointing interface to provide character data, wherein the shared language model is adaptable by a correction process comprising:

specifying, responsive to freehand input, a selection of text generated by the message recognizer;

initiating a correction mode for modifying the selection;

providing a list of alternative text comparable to the selection;

updating the list based on character data communicated after initiation of the correction mode;

modifying the selection responsive to character data; and

adapting the shared language model in accordance with the modification to the selection.

10. The system of claim 1 wherein:

(a) the speech recognizer provides text data in accordance with at least one model; and

(b) the speech recognizer is responsive to the indicia of editing commands to adapt the model.

11. The system of claim 10 further comprising a character interface responsive to character input communicated via the pointing interface to provide character data, wherein the speech recognizer provides text data in accordance with at least a syntactic model that is adaptable by a correction process comprising:

specifying, responsive to freehand input, a selection of text generated by the speech recognizer;

initiating a correction mode for modifying the selection;

providing a list of alternative text comparable to the selection;

updating the list based on character data communicated after initiation of the correction mode;

modifying the selection responsive to character data; and

adapting the model in accordance with the modification to the selection.

12. The system of claim 10 wherein:

(c) the speech recognizer provides text data in accordance with at least a semantic model, the model comprising a plurality of semantic clusters; and

(d) the semantic model is adaptable by a process comprising;

listing a plurality of text segments having a low probability of shared cluster occurrence;

selecting, responsive to freehand input, at least one text segment of the plurality; and

adapting the semantic model to bias against text segments having a high probability of shared cluster occurrence with each selected text segment.

13. The system of claim 10 wherein:

(a) the speech recognizer provides text data in accordance with at least a semantic model, the semantic model comprising a plurality of semantic clusters; and

(b) the semantic model is adaptable by a process comprising;

generating a maximum likelihood text segment responsive to voice input;

listing at least one alternative text segment, the maximum likelihood text segment and each alternative text segment forming a plurality of text segments, each text segment of the plurality having a high probability of global cluster occurrence and a low probability of shared cluster occurrence;

selecting, responsive to freehand input, at least one text segment of the plurality; and

adapting the semantic model to apply a negative bias against a tested text segment, the severity of the bias being proportional to the probability of shared cluster occurrence between the tested text segment and the selected text segment or segments.

14. The system of claim 13 wherein the bias is inversely proportional to the probability of global cluster occurrence of the tested text segment.

15. The system of claim 13 wherein the bias is inversely proportional to the amount of text data separating the tested text segment and the selected text segment or segments.

16. The system of claim 1 wherein the stylus further comprises an identification circuit for providing, to the speech recognizer, an identification of an owner of the stylus.

17. The system of claim 1 wherein the stylus further comprises an activation circuit for automatically activating the speech recognizer.

18. The system of claim 17 wherein the activation circuit comprises:

(a) a first sound sensor responsive to sound impinging on the stylus within a first angular range to provide a first sensing signal;

(b) a second sound sensor responsive to sound impinging on the stylus outside the first angular range to provide a second sensing signal; and

(c) a comparator for comparing the first and second sensing signals.

19. The system of claim 17 wherein the activation circuit comprises a proximity sensor, responsive to proximity of the microphone to a human head, for controlling activation of the speech recognizer by the activation circuit.

20. The system of claim 19 wherein the proximity sensor comprises a radiation sensor responsive to field radiation impinging on the stylus.

21. The system of claim 20 wherein the radiation sensor is responsive to electromagnetic field radiation in the infrared spectrum.

22. The system of claim 20 wherein:

(a) the radiation sensor is responsive to field radiation impinging on the stylus within a first angular range to provide a first sensing signal and within a second angular range to provide a second sensing signal; and

(b) the proximity sensor further comprises a comparator for comparing the first and second sensing signals.

23. The system of claim 22 wherein:

(a) the activation circuit is configured to activate the speech recognizer when the microphone is at a predetermined proximity to the human head, wherein the microphone is;

(1) oriented toward the human head within a predetermined angular range; and

(2) displaced from the human head within a predetermined separation range;

(b) the second angular range falls completely within the first angular range; and

(c) the predetermined proximity is determined by all of at least two conditions being true, wherein the conditions are;

(1) the amplitude of the second sensing signal reaching a first predetermined threshold; and

(2) the ratio between the amplitude of the second sensing signal and the amplitude difference between the first and second sensing signals reaching a second predetermined threshold.

24. The system of claim 20 wherein the radiation sensor is responsive to acoustic field radiation, the system further comprising:

(a) an ultrasonic transmitter for sending acoustic field radiation to the human head, such that the radiation sensor provides a time-varying signal descriptive of reflected acoustic field radiation impinging on the stylus from the human head;

(b) a signal processor responsive to the time-varying signal so as to provide a distance profile of the acoustic field radiation; and

(c) a comparator for comparing the distance profile to an expected profile, the expected profile corresponding to acoustic field radiation expected to impinge on the stylus from the human head when the microphone is at a predetermined proximity to the human head.

25. A system responsive to voice input and freehand input to generate and edit text, the system comprising:

(a) a stylus comprising;

(1) a pointer for communicating freehand input; and

(2) a microphone for receiving voice input;

(b) at least one speech recognizer responsive to voice input to provide text data;

(c) a plurality of tablet interfaces, each including a respective tablet surface and being responsive to freehand input to provide indicia of editing commands;

(d) a plurality of displays, each being integrated into a respective tablet surface; and

(e) at least one editor having access to the text data, responsive to the indicia of editing commands to modify the text data as specified by the editing commands.

26. The system of claim 25 further comprising:

(a) a communications link interconnecting the displays; and

(b) a command interpreter for controlling the communications link, the command interpreter being responsive to the indicia of editing commands to transfer text data between the displays as specified by the editing commands.

27. A method for processing text, the method comprising:

providing a stylus comprising a pointing interface and a microphone;

providing a tablet responsive to the pointing interface;

generating text by dictating into the microphone; and

editing text by manually manipulating the pointing interface on the tablet.

28. The method of claim 27 further comprising:

moving the stylus across visual indicia of dictated text that has been erroneously generated to specify a selection from the text; and

correcting the selection.

29. The method of claim 28 further comprising:

generating text in accordance with an adaptable language model; and

adapting the language model responsive to the correction of the selection.

30. The method of claim 29 further comprising:

providing a character interface responsive to the pointing interface;

providing a list of alternative text comparable to the selection;

updating the list based on character input communicated after specification of the selection;

modifying the selection responsive to character input communicated via the pointing interface; and

adapting the language model in accordance with the modification to the selection.

31. The method of claim 29 wherein the language model comprises a syntactic model and a semantic model, the semantic model comprising a plurality of semantic clusters, the method further comprising:

listing a plurality of text segments having a low probability of shared cluster occurrence;

selecting, responsive to freehand input, at least one text segment of the plurality; and

adapting the semantic model to bias against text segments having a high probability of shared cluster occurrence with each selected text segment.

32. The method of claim 29 wherein the language model comprises a syntactic model and a semantic model, the semantic model comprising a plurality of semantic clusters, the method further comprising:

generating a maximum likelihood text segment responsive to voice input;

listing at least one alternative text segment, the maximum likelihood text segment and each alternative text segment forming a plurality of text segments, each text segment of the plurality having a high probability of global cluster occurrence and a low probability of shared cluster occurrence;

selecting, responsive to freehand input, at least one text segment of the plurality; and

adapting the semantic model to apply a negative bias against a tested text segment, the severity of the bias being proportional to the probability of shared cluster occurrence between the tested text segment and the selected text segment or segments.

33. The method of claim 32 wherein the bias is inversely proportional to the probability of global cluster occurrence of the tested text segment.

34. The method of claim 32 wherein the bias is inversely proportional to the amount of text data separating the tested text segment and the selected text segment or segments.

35. The method of claim 27 further comprising sending an identification signal to the speech recognizer to identify the stylus.

36. The method of claim 27 further comprising activating the speech recognizer by moving the stylus within a predetermined proximity of a human head.

37. A stylus comprising:

(a) a stylus body having a proximal end and a distal end;

(b) a pointing interface toward the distal end;

(c) an identification circuit for providing an identification signal; and

(d) a wireless transmitter for transmitting an electromagnetic signal comprising the voice signal and the identification signal.

38. The stylus of claim 37 further comprising a microphone toward the proximal end for providing a voice signal responsive to voice input, wherein the electromagnetic signal further comprises the voice signal.

39. The stylus of claim 38 wherein the identification signal and the voice signal are time-division multiplexed on the electromagnetic signal.

40. The stylus of claim 37 wherein the microphone is responsive to sound impinging on the stylus from the direction of the proximal end with a first sensitivity and from the direction of the distal end with a second sensitivity, the first sensitivity being significantly greater than the second sensitivity.

41. The stylus of claim 37 further comprising an activation circuit for automatically enabling the voice signal.

42. The stylus of claim 41 wherein the activation circuit comprises a proximity sensor, responsive to proximity of the microphone to a human head, for controlling enabling of the voice signal by the activation circuit.

43. A stylus comprising:

(a) a stylus body having a proximal end and a distal end;

(b) a pointing interface toward the distal end;

(c) a microphone toward the proximal end for providing a voice signal responsive to voice input;

(d) an activation circuit for automatically enabling the voice signal; and

(e) a wireless transmitter for transmitting an electromagnetic signal comprising the voice signal.

44. The system of claim 43 wherein the activation circuit comprises:

(a) a first sound sensor responsive to sound impinging on the stylus within a first angular range to provide a first sensing signal;

(b) a second sound sensor responsive to sound impinging on the stylus outside the first angular range to provide a second sensing signal; and

(c) a comparator for comparing the first and second sensing signals.

45. The system of claim 43 wherein the activation circuit comprises a proximity sensor, responsive to proximity of the microphone to a human head, for controlling the enabling of the voice signal.

46. The system of claim 45 wherein the proximity sensor comprises a radiation sensor responsive to field radiation impinging on the stylus.

47. The system of claim 46 wherein the radiation sensor is responsive to electromagnetic field radiation in the infrared spectrum.

48. The system of claim 46 wherein:

(a) the radiation sensor is responsive to field radiation impinging on the stylus within a first angular range to provide a first sensing signal and within a second angular range to provide a second sensing signal; and

(b) the proximity sensor further comprises a comparator for comparing the first and second sensing signals.

49. The system of claim 48 wherein:

(a) the activation circuit is configured to enable the voice signal when the microphone is at a predetermined proximity to the human head, wherein the microphone is;

(1) oriented toward the human head within a predetermined angular range; and

(2) displaced from the human head within a predetermined separation range;

(b) the second angular range falls completely within the first angular range; and

(c) the predetermined proximity is determined by all of at least two conditions being true, wherein the conditions are;

(1) the amplitude of the second sensing signal reaching a first predetermined threshold; and

(2) the ratio between the amplitude of the second sensing signal and the amplitude difference between the first and second sensing signals reaching a second predetermined threshold.

50. The system of claim 46 wherein the radiation sensor is responsive to acoustic field radiation, the system further comprising:

(a) an ultrasonic transmitter for sending acoustic field radiation to the human head, such that the radiation sensor provides a time-varying signal descriptive of reflected acoustic field radiation impinging on the stylus from the human head;

(b) a signal processor responsive to the time-varying signal so as to provide a distance profile of the acoustic field radiation; and

(c) a comparator for comparing the distance profile to an expected profile, the expected profile corresponding to acoustic field radiation expected to impinge on the stylus from the human head when the microphone is at a predetermined proximity to the human head.

51. A method for message recognition in accordance with at least a semantic language model, the method comprising performing a plurality of hypothesis tests between a plurality of text segments and at least one stop segment, each hypothesis test including computing a probability according to the equation:

52. In a message recognizer, a semantic language model comprising a plurality of semantic clusters, the language model being adaptable by a process comprising the steps of:

(a) listing a plurality of text segments;

(b) selecting, responsive to a first type of input, at least one text segment of the plurality; and

(c) adapting the semantic model to bias against text segments having a high probability of shared cluster occurrence with each selected text segment.

53. The language model of claim 52 wherein the listed text segments have a low probability of shared cluster occurrence.

54. The language model of claim 52 wherein:

the context identification further comprises generating a maximum likelihood text segment responsive to a second type of input;

said step (a) further comprises listing at least one alternative text segment, the maximum likelihood text segment and each alternative text segment forming a plurality of text segments, each text segment of the plurality having a high probability of global cluster occurrence and a low probability of shared cluster occurrence;

said step (b) further comprises selecting, responsive to the first type of input, at least one text segment of the plurality; and

said step (c) further comprises adapting the semantic model to apply a negative bias against a tested text segment, the severity of the bias being proportional to the probability of shared cluster occurrence between the tested text segment and the selected text segment or segments.

55. The language model of claim 54 wherein:

(a) the first type of input is freehand input; and

(b) the second type of input is voice input.

56. The language model of claim 54 wherein the bias is inversely proportional to the probability of global cluster occurrence of the tested text segment.

57. The language model of claim 54 wherein the bias is inversely proportional to the amount of text data separating the tested text segment and the selected text segment or segments.

58. A method for adapting a semantic language model comprising a plurality of semantic clusters, the method comprising the steps of:

(a) listing a plurality of text segments having a low probability of shared cluster occurrence;

(b) selecting, responsive to freehand input, at least one text segment of the plurality; and

(c) adapting the semantic model to bias against text segments having a high probability of shared cluster occurrence with each selected text segment.

59. The method of claim 58 wherein:

the context identification further comprises generating a maximum likelihood text segment responsive to voice input;

said step (a) further comprises listing at least one alternative text segment, the maximum likelihood text segment and each alternative text segment forming a plurality of text segments, each text segment of the plurality having a high probability of global cluster occurrence and a low probability of shared cluster occurrence;

said step (b) further comprises selecting, responsive to freehand input, at least one text segment of the plurality; and

said step (c) further comprises adapting the semantic model to apply a negative bias against a tested text segment, the severity of the bias being proportional to the probability of shared cluster occurrence between the tested text segment and the selected text segment or segments.

60. The method of claim 59 wherein the bias is inversely proportional to the probability of global cluster occurrence of the tested text segment.

61. The method of claim 59 wherein the bias is inversely proportional to the amount of text data separating the tested text segment and the selected text segment or segments.

62. A system for generating text responsive to message input of a first type and a second type, the system comprising:

(a) a unified message model including;

(1) a shared language model;

(2) a first model specific to a first type of message recognition; and

(3) a second model specific to a second type of message recognition;

(b) a first message recognizer, responsive to the first type of message input to provide text data in accordance with both the shared language model and the first model; and

(c) a second message recognizer, responsive to the second type of message input to provide text data in accordance with both the shared language model and the second model.

63. The system of claim 62 wherein:

(a) the first message recognizer is a speech recognizer, the first type of message input is voice input, and the first model is an acoustic model; and

(b) the second message recognizer is a handwriting recognizer, the second type of message input is freehand input, and the second model is a handwriting model.

64. The system of claim 62 wherein the shared language model comprises a syntactic model and a semantic model.

65. The system of claim 62 wherein the message model is a local message model, and wherein the system further comprises a network node for establishing a network connection to a user message model such that the local message model may be initialized to be in conformance with the user message model.

66. The system of claim 65 wherein the network node connects to a wide area network.

67. A method for recognizing a message input from a specific user, the method comprising:

providing;

(1) a local message model responsive to initialization by a user message model;

(2) a network including a first node and a second node, the first node being proximate the user and the second node being proximate the user message model; and

identifying the specific user to establish a connection on the network between the local message model and the user message model;

transferring data on the connection to modify the local message model to be in conformance with the user message model; and

recognizing message input from the user in accordance with the initialized local message model.

68. The method of claim 67 further comprising:

after modifying the local message model to be in conformance with the user message model, adapting the local message model in accordance with editing commands from the user; and

transferring data on the connection to modify the user message model to be in conformance with the adapted local message model.

69. The method of claim 67 wherein the local message model is implemented in a speech recognizer that is proximate the first node.

70. The method of claim 67 wherein:

(a) the message input comprises voice input; and

(b) the local message model comprises a language model and an acoustic model.

71. The method of claim 67 wherein:

(a) the message input comprises freehand input; and

(b) the local message model comprises a language model and a handwriting model.

72. The method of claim 67 wherein:

(a) the message input comprises voice input and freehand input; and

(b) the local message model comprises a language model, an acoustic model, and a handwriting model.

73. The system of claim 67 wherein the specific user is identified by an identification circuit in a stylus employed by the user to convey message input.

74. The system of claim 67 wherein the network is a wide area network.

75. A system responsive to voice input and freehand input to generate and edit text, the system comprising:

(a) a stylus comprising;

(1) a pointer for communicating freehand input; and

(2) a microphone for receiving voice input;

(b) means for generating text data responsive to voice input; and

(c) means for editing the text data responsive to freehand input.

76. A method for displaying a document, the method comprising:

displaying a first view of the document on a first display;

displaying a second view of the document on a second display; and

upon modification of the document, modifying the first and second views as needed to reflect the modification.

77. The method of claim 76 wherein:

(a) the first display is integrated into a tablet surface of a first tablet;

(b) the second display is integrated into a tablet surface of a second tablet; and

(c) the first and second tablets are coupled together such that;

(1) modification made to the document by user input to the first tablet is communicated to the second tablet to modify the second view as needed to reflect the modification; and

(2) modification made to the document by user input to the second tablet is communicated to the first tablet to modify the first view as needed to reflect the modification.

78. A text processing system comprising:

(a) a stylus having a proximal end and a distal end, the stylus comprising;

(1) a pointer toward the distal end for communicating freehand input;

(2) a microphone toward the proximal end for receiving voice input;

(3) an activation circuit for automatically activating the speech recognizer; and

(4) an identification circuit for providing, to the speech recognizer, an identification of the stylus;

(b) a tablet interface, responsive to freehand input, comprising a plurality of tablet surfaces;

(c) a plurality of self-contained enclosures, each of the enclosures comprising;

(1) a front face having a height and a width, the front face comprising a respective one of the tablet surfaces; and

(2) a back face, separated from the front face by a depth, the depth being significantly less than either the height or the width;

(d) a display comprising a plurality of displays, each of the displays being integrated into a respective tablet surface;

(e) a wireless transmitter and a wireless receiver for effecting a communications link to interconnect the displays;

(f) a command interpreter for controlling the communications link, the command interpreter being responsive to command indicia communicated via freehand input to transfer text between the displays; and

(g) a speech recognizer responsive to voice input to generate text in accordance with a language model specific to an owner of the stylus, for display on the display.