Message recognition using shared language model

• US 8,204,737 B2
• Filed: 03/22/2005
• Issued: 06/19/2012
• Est. Priority Date: 07/17/1999
• Status: Expired due to Term

• Current Assignee: Buffalo Patents, LLC
• Original Assignee: Optical Research Partners LLC (Intellectual Ventures LLC)
• Inventors: Suominen, Edwin A.
• Primary Examiner(s): COLUCCI, MICHAEL C
• Application Number:
  US11/088,032
• Publication Number:
  US 20050171783A1
• Time in Patent Office: 2,646 Days
• Field of Search: 382/187, 382/124, 704/270, 704/256, 704/235, 704/257, 704/270.1, 379/56.1, 455/550.1, 318/568.11
• US Class Current: 704/9
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

A system is disclosed for generating text responsive to both voice and handwriting input, including a microphone, stylus, and a tablet having an flat-panel display integrated into its surface. The system performs speech and handwriting recognition using a shared language model, which in one embodiment is trainable responsive to user correction of errors in either speech or handwriting recognition. Various other systems and methods are also disclosed.

Claims

1. A system for generating text responsive to voice and handwriting input, comprising:

a tablet surface, wherein the system is configured to receive freehand input based at least in part on manipulation of a stylus relative to the tablet surface; and

a microphone for receiving voice input;

wherein the system is configured to;

select a particular user message model from a plurality of user message models;

adjust a language model of a local message model based at least in part on the particular user message model, wherein the local message model includes the language model, a handwriting model, and an acoustic model;

generate text data based at least in part on the freehand input, the adjusted language model, and the handwriting model; and

generate text data based at least in part on the voice input, the adjusted language model, and the acoustic model.

2. The system of claim 1, wherein the system is further configured to:

identify information relating to a particular user;

wherein the particular user message model is selected based at least in part on the information relating to the particular user; and

,

wherein the plurality of user message models includes two or more user message models corresponding to users that are different from the particular user.

3. The system of claim 2 wherein the microphone is housed in the stylus.

4. The system of claim 1, wherein the system is further configured to train the particular user message model by updating the acoustic model.

5. The system of claim 1, further comprising a proximity sensor configured to sense a proximity of a user to the microphone.

6. The system of claim 1, further comprising an identification circuit configured to identify an owner of the stylus, wherein the system is configured to select the particular user message model at least in part in response to the identification circuit.

7. The system of claim 1, wherein the system is configured to select the particular user message model based at least in part on an identifying signal received from the microphone.

8. The system of claim 7, wherein the system is configured to access the particular user message model via a wide area network.

9. The system of claim 1, wherein the system adjusting the language model includes adjusting a syntactic model and a semantic model included in the-language model.

10. The system of claim 9, wherein the semantic model comprises at least one semantic cluster selected from the group consisting of:

a semantic cluster of a first type that is a mathematically derived group of vectors representing text segments;

a semantic cluster of a second type that is a discrete grouping of text having a particular context; and

a semantic cluster of a third type that is defined with respect to an individual reference word that is indicative of a particular context.

11. The system of claim 1, further comprising an activation circuit configured to sense movement of the stylus toward the tablet surface, wherein the system is further configured to receive the freehand input in response to the activation circuit sensing the movement.

12. The system of claim 1, wherein the particular user message model includes user-specific acoustic model information and user-specific handwriting model information.

13. A method, comprising:

receiving, at a device, freehand input based at least in part on manipulation of a stylus;

receiving, at the device, audio input based at least in part on information received at a microphone;

selecting, via the device, a selected user message model from a plurality of user message models;

adjusting, via the device, a language model of a local message model, wherein the local message model includes the language model, a handwriting model, and an acoustic model, wherein the adjusting the language model is based at least in part on the selected user message model;

generating, by the device, converted handwriting text data based at least in part on the freehand input, the adjusted language model, and the handwriting model; and

generating, by the device, converted speech text data based at least in part on the audio input, the adjusted language model, and the acoustic model.

14. The method according to claim 13, further comprising switching from receiving audio input to receiving freehand input in response to detecting movement of the stylus towards a tablet.

15. The method according to claim 13, further comprising accessing the selected user message model via a network.

16. The method according to claim 13, further comprising training the selected user message model.

17. The method according to claim 13, further comprising training the local message model by adjusting the acoustic model of the local message model and leaving the language model of the local message model unchanged.

18. The method according to claim 13, further comprising training the local message model by adjusting the handwriting model of the local message model and leaving the language model of the local message model unchanged.

19. An article of manufacture including a non-transitory computer-readable medium having stored thereon instructions executable by a computing device to cause the computing device to perform operations comprising:

receiving freehand input based at least in part on manipulation of a stylus;

receiving audio input based at least in part on information received at a microphone;

selecting a particular user message model from a plurality of user message models;

adjusting a language model of a local message model based at least in part on the particular user message model, the local message model including the language model, a handwriting model and an acoustic model;

generating converted handwriting text data based at least in part on the freehand input, on the adjusted language model, and on the handwriting model; and

generating converted speech text data based at least in part on the audio input, on the adjusted language model, and on the acoustic model.

20. An apparatus, comprising:

means for receiving freehand input based at least in part on manipulation of a stylus;

means for receiving audio input based at least in part on audio information;

means for selecting a particular user message model from a plurality of user message models;

means for adjusting a language model of a local message model based at least in part on the particular user message model, the local message model including the language model, a handwriting model, and an acoustic model;

means for generating converted handwriting text data based at least in part on the freehand input, on the adjusted language model, and on the handwriting model; and

means for generating converted speech text data based at least in part on the audio input, on the adjusted language model, and on the acoustic model.