Processing handwritten and hand-drawn input and speech input

• US 6,438,523 B1
• Filed: 05/18/1999
• Issued: 08/20/2002
• Est. Priority Date: 05/20/1998
• Status: Expired due to Fees

• Current Assignee: Fonix Corp.
• Original Assignee: John A. Oberteuffer, John Wilbanks, Kyung-Ho Loken-Kim, William Kania
• Inventors: Kania, William, Oberteuffer, John A., Loken-Kim, Kyung-Ho, Wilbanks, John
• Primary Examiner(s): Knepper, David D.
• Application Number:
  US09/313,664
• Time in Patent Office: 1,190 Days
• Field of Search: 704/235, 704/236, 704/255, 704/270, 704/260, 704/251, 382/187, 382/186, 382/189, 345/157
• US Class Current: 704/270
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 2015/228 of application context

Abstract

Techniques consistent with this invention process handwritten or hand-drawn input and speech input. Method steps include recognizing received handwritten or hand-drawn input, recognizing received speech input, and creating or modifying an electronic document according to the speech or handwritten or hand-drawn input. An apparatus includes structure for recognizing handwritten or hand-drawn input, structure for recognizing speech input, and structure for activating modes for processing the handwritten or hand-drawn input and the speech input responsive to commands in the handwritten or hand-drawn input and the speech input.

Claims

1. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first operating mode and a second operating mode before processing handwritten input, hand-drawn input, or speech input;

recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

2. The method of claim 1, wherein the step of switching said first operating mode or said second operating mode includes the substep of switching either said first mode or said second mode in response to a gesture.

3. The method of claim 1, wherein the step of switching said first operating mode or said second operating mode includes the substep of switching either said first mode or said second mode in response to speech input.

4. The method of claim 1, wherein the step of switching said first operating mode or said second operating mode includes the substep of switching either said first mode or said second mode in response to a mechanical switch activated by an operator.

5. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as text and said handwritten input is recognized as text; and

switching processing modes between said first combination mode and said second combination mode.

6. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a command; and

switching processing modes between said first combination mode and said second combination mode.

7. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as text; and

switching processing modes between said first combination mode and said second combination mode.

8. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text and said handwritten input is recognized as text;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a command; and

switching processing modes between said first combination mode and said second combination mode.

9. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text and said handwritten input is recognized as text;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as text; and

switching processing modes between said first combination mode and said second combination mode.

10. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as text;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a command; and

switching processing modes between said first combination mode and said second combination mode.

11. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

12. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

13. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text and said handwritten input is recognized as text;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

14. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text and said handwritten input is recognized as text;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

15. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a graphic;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a command; and

switching processing modes between said first combination mode and said second combination mode.

16. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a graphic;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as text; and

switching processing modes between said first combination mode and said second combination mode.

17. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as text input and said handwritten input is recognized as a graphic;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

18. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a command;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

19. A method for processing handwritten input or hand-drawn input and speech input comprising the steps, performed by a computer, of:

selecting a first combination mode and a second combination mode before processing handwritten, hand-drawn, or speech input;

processing said handwritten input or said speech input in said first combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as text input;

processing said handwritten input or said speech input in said second combination mode wherein said speech input is recognized as a command and said handwritten input is recognized as a graphic; and

switching processing modes between said first combination mode and said second combination mode.

20. A pen tablet computer for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

21. The pen tablet computer of claim 20 further comprising a pen-integrated microphone.

22. A palm computer for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

23. The palm computer of claim 22 further comprising a pen-integrated microphone.

24. A kiosk computer system for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

25. The kiosk computer system of claim 24, wherein said means for recognizing the handwritten or hand-drawn input includes either a touch screen, a digitizing tablet, or a video gesture recognizer.

26. A computer for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

27. The computer of claim 26, wherein said means for recognizing the handwritten or hand-drawn input includes either a touch screen, a digitizing tablet, a three-dimensional wireless mouse, or a video gesture recognizer.

28. The computer of claim 26 further comprising

a display with a background responsive to an operating state of either said means for recognizing the speech input or said means for recognizing the handwritten or hand-drawn input.

29. The display of claim 28 further comprising

means for changing screen tint.

30. The display of claim 28 further comprising

means for displaying lines.

31. A television or video cassette recorder remote control for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

32. The television or video cassette recorder remote control of claim 31, wherein said means for recognizing the handwritten or hand-drawn input includes either a three-dimensional wireless mouse or a video gesture recognizer.

33. An electronic whiteboard for processing handwritten input or hand-drawn input and speech input comprising:

selecting a first operating mode and a second operating mode before processing handwritten, hand-drawn, or speech input;

means for recognizing the speech input as either a command or text for an electronic document in accordance with said first operating mode;

means for recognizing the handwritten or hand-drawn input as a command, text for an electronic document, or a graphic for an electronic document in accordance with said second operating mode; and

means for switching said first operating mode or said second operating mode to change how the computer recognizes said speech input or said handwritten or hand-drawn input.

34. The electronic whiteboard of claim 33, wherein said means for recognizing the handwritten or hand-drawn input includes either a three-dimensional wireless mouse, a pen, a touch screen, or a video gesture recognizer.

35. A computer for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or hand-drawn input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten or hand-drawn input as a command.

36. A pen tablet computer for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or hand-drawn input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten or hand-drawn input as a command.

37. A palm computer for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or handwritten input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or handwritten input recognizes handwritten or hand-drawn input as a command.

38. A telephone for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or hand-drawn input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten or hand-drawn input as a command.

39. A television or video cassette recorder remote control for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or hand-drawn input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten or hand-drawn input as a command.

40. An electronic whiteboard for processing handwritten input or hand-drawn input and speech input comprising:

means for recognizing handwritten or hand-drawn input;

means for recognizing speech input; and

a mechanical switch for switching among operating modes, wherein said operating modes include

a first mode wherein said means for recognizing speech input recognizes speech as text,

a second mode wherein said means for recognizing speech input recognizes speech as a command,

a third mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten input as text,

a fourth mode wherein said means for recognizing handwritten or hand-drawn input recognizes hand-drawn input as a graphic, and

a fifth mode wherein said means for recognizing handwritten or hand-drawn input recognizes handwritten or hand-drawn input as a command.