Virtual invisible keyboard

• US 7,042,442 B1
• Filed: 06/27/2000
• Issued: 05/09/2006
• Est. Priority Date: 06/27/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kanevsky, Dimitri, Zlatsin, Alexander, Sabath, Mariusz
• Primary Examiner(s): Wu, Xiao
• Assistant Examiner(s): LESPERANCE, JEAN E
• Application Number:
  US09/603,980
• Time in Patent Office: 2,142 Days
• Field of Search: 345/168, 345/169, 345/173, 345/157, 345/158, 345/163, 345/156, 345/159, 178 1801- 1803, 178/18.04, 178/18.05, 178/18.06, 178/18.07
• US Class Current: 345/169
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

The invention uses a recognition system of gestures that maps sequences of gestures to keys strings. In the practice of this invention, a user produces gestures without keyboards. Many experienced typists can type without looking at keyboards, and typists can make gestures, in the absence of a keyboard, that are similar to gestures that would be made if there were a keyboard. The gesture recognition system captures gestures for example, (via cameras) and interprets them as pressing an invisible keyboards, as if a keyboard were actually placed in a certain location under the typists hands. To coordinate the invisible keyboard in the correct place under the hands, the user may be provided with feedback. He or she can either view the results of the gestures via a display or hear sounds, via speakers, indicating the results of the gestures.

Claims

1. An information input processing computer system for mapping gestures to keys of an invisible virtual keyboard, the system comprising one or several cameras, one or more memories with CPU connected to the cameras, and processes running in the CPU that associate gesture movements made without touching any touch sensors with typing and produce gesture associated textual output, wherein said processes capture gesture images, classify the gesture images into classes, associate each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, assign a probability to each of the possible keys associated with the class, and integrate the probabilities assigned to the possible keys to identify a word for a sequence of gestures;

and

wherein the processes that integrate the probabilities use one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

2. The gesture-key mapping system as in claim 1, where a feedback is provided to the user on what kind of keys are associated with the user'"'"'s gestures.

3. The gesture-key mapping system as in claim 2, where the feedback is provided using one or more of the following:

displaying keys on a display, playing sounds labels for keys, displaying image indicators on a display, playing special sound indicators, projecting the keyboard to any surface, and displaying picture of the keyboard with user'"'"'s hands.

4. An information input processing, gesture-key mapping computer system for generating text from hand gestures of a user relative to an invisible virtual keyboard, the system comprising one or several cameras, one or more memories with CPU connected to the cameras, and processes running in the CPU that associate gesture movements with typing and produce gesture associated textual output, where the gesture-key processing is provided using the following modules:

a) a gesture capturing module that captures gestures, relative to an invisible keyboard and made without touching any touch sensors, through camera sensors;

b) a gesture classificator module that classifies the gestures into;

c) an associator module for associating each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, assign a probability to each of the possible keys associated with the class; and

d) an integrator module that integrates the probabilities assigned to the possible keys to identify a word for a sequence of gestures; and

wherein the integrator module includes one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

5. A system according to claim 4, wherein each sequence of keys receives a probability score, and the sequences of keys that receive low scores are removed and are not continuing when new candidates for keys arrive.

6. A system according to claim 4, further comprising a gesture correlator module to allow to adjust automatically the invisible keyboard to changes in the hand positions of the user, and wherein the gesture correlator module acts between the gesture classificator module and the associator module to maintain a consistent mapping of gesture classes to keys of the invisible keyboard despite said changes in the hand positions of the user.

7. The method for producing a textual output in which a user makes typing like gestures relative to an invisible virtual keyboard made without touching any touch sensors and without the presence of a real keyboard and the gestures are associated with the most probable keys that would be typed if a keyboard were presented, said method including the steps of using a computer system to map gestures made, without touching any touch sensors, to keys of the virtual keyboard, including the steps of running processes on the computer to capture gesture images, to classify the gesture images into classes, to associate each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, to assign a probability to each of the possible keys associated with the class, and to integrate the probabilities assigned to the possible keys to identify a word for a sequence of gestures;

and

wherein the processes to integrate the probabilities use one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

8. The method for producing a textual output in which a user makes typing like gestures relative to an invisible virtual keyboard and without the presence of a real keyboard and the gestures are associated with the most probable keys that would be typed if a keyboard were presented, said method including the step of using a computer system to map gestures made, without touching any touch sensors, to keys of the virtual keyboard, including the step of running processes on the computer to capture gesture images, to classify the gesture images into classes, to associate each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, to assign a probability to each of the possible keys associated with the class, and to integrate the probabilities assigned to the possible keys to identify a word for a sequence of gestures, and wherein the probability is computed using HMM;

and

wherein the processes to integrate the probabilities use one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

9. A method of typing using a virtual keyboard having a multitude of virtual keys, comprising the steps:

making typing gestures relative to an invisible virtual keyboard made without touching any touch sensors and without any real keyboard;

sensing the typing gestures; and

producing, from the sensed typing gestures, gesture associated textual output including the step of running processes on a computer to capture gesture images, to classify the gesture images into classes, to associate each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, to assign a probability to each of the possible keys associated with the class, and to integrate the probabilities assigned to the possible keys to identify a word for a sequence of gestures; and

wherein the processes to integrate the probabilities use one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

10. A method according to claim 9, wherein the typing gestures are made by a person, and further comprising the steps of providing feedback to the person on texture output associated with the gestures.

11. A method according to claim 10, wherein the step of providing feedback includes the step of displaying an image of typing keys associated with the gestures.

12. A method according to claim 9, wherein the step of running processes on the computer includes the step of mapping the sensed typing gestures to keys of the keyboard based on a statistical machine that interprets sequences of typing like gesture classes as intended words based on user typing gesture models.

13. A method according to claim 9, wherein gesture input is represented as wave forms that are digitized and clustered in gesture frames, and then processed by HMM machines that represent difference models for different typing patterns;

and the step of running processes includes the step of using two statistical components, a first statistical component to map gestures to keys, and a second, language model component to map keys to letters and words.

14. A method of typing using an invisible virtual keyboard, comprising the steps:

making typing gestures relative to an invisible virtual keyboard made without touching any touch sensors and without any real keyboard;

sensing the typing gestures; and

producing, from the sensed typing gestures, gesture associated textual output; and

wherein the producing step includes the steps of classifying the gestures into classes, associating each of said classes with one or more possible keys of the invisible keyboard, for each of said classes, assign a probability to each of the possible keys associated with the class, and integrating the probabilities assigned to the possible keys to identifying a word for a sequence of gestures; and

wherein the step of integrating the probabilities includes the step of using one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

15. A method according to claim 14, wherein the producing step further includes the step of associating gesture classes with individual typing keys.

16. A method according to claim 14, further comprising providing training data in words or sentences with certain timing data.

17. A typing system using an invisible keyboard, comprising:

means for sensing typing gestures made relative to an invisible virtual keyboard made without touching any touch sensors and without any real keyboard; and

means for producing, from the sensed typing gestures, gesture associated textual output said producing means including a computer and processes running on the computer to capture gesture images, to classify the gesture images into classes, and to associate each of the classes with one or more possible keys of the invisible keyboard, for each of said classes, to assign a probability to each of the possible keys associated with the class, and to integrate the probabilities assigned to the possible keys to identify a word for a sequence of gestures; and

wherein the processes to integrate the probabilities use one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

18. A system according to claim 17, wherein the typing gestures are made by a person, and further comprising means for providing feedback to the person on texture output associated with the gestures.

19. A system according to claim 18, wherein the means for providing feedback includes means for displaying an image of typing keys associated with the gestures.

20. A typing system using an invisible virtual keyboard, comprising:

means for sensing typing gestures made relative to a virtual, invisible keyboard made without touching any touch sensors and without any real keyboard; and

means for producing, from the sensed typing gestures, gesture associated textual output; and

wherein the producing means includes means for classifying the gestures into classes, for associating each of said classes with one or more possible keys of the invisible keyboard, for each of said classes, for assigning a probability to each of the possible keys associated with the class, and for integrating the probabilities assigned to the possible keys to identifying a word for a response of gestures; and

wherein the means for integrating the probabilities includes one or more of the following;

a) language module component that estimate probabilities of word strings corresponding to key candidate sequences;

b) character frequency module that estimate probabilities of character strings corresponding key candidate sequences;

c) confusable matrix that estimate how often correct gesture classes are confusable with another gesture classes;

d) gesture classes probability model that estimate probability of observing a string of gesture classes given a sequence of gesture frames;

e) computation of a probability of production a sequence of keys given a string of gesture frames;

f) generation of a lattice of sequences of keys given sequence of gesture frames;

g) finding the most probable sequence of keys from the lattice of key candidate strings.

21. A system according to claim 20, wherein the producing means further includes means for associating gesture classes with individual typing keys.