Techniques For Using Gesture Recognition To Effectuate Character Selection

• US 20150363001A1
• Filed: 05/20/2015
• Published: 12/17/2015
• Est. Priority Date: 06/13/2014
• Status: Active Grant

• Current Assignee: Thomas Malzbender
• Original Assignee: Thomas Malzbender
• Inventors: Malzbender, Thomas
• Granted Patent: US 10,146,318 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

This disclosure provides a pose- or gesture-based recognition system that processes images of the human hand, downconverts degrees of freedom of the human hand to lower-dimensional space, and then maps the downconverted space to a character set. In one embodiment, the system is implemented in a smart phone or as a computer-input device that uses a virtual keyboard. As the user moves his or her hand, the smart phone or computer provides simulated vocal feedback, permitting the user to adjust hand position or motion to arrive at any desired character; this is particularly useful for embodiments which use a phonetic character set. Software that performs character selection can be implemented in a manner that is language/region agnostic, with a contextual dictionary being used to interpret a phonetic character set according to a specific language or region.

Claims

1. A method, comprising:

receiving with a computing device digital data representing a string of images, the images representing at least part of a user;

processing the data using software to detect therefrom a string of gestures of the user, each gesture represented as a vector;

mapping the vectors to a string of phonetic elements; and

identifying one or more words in a written language corresponding to a string of the phonetic elements.

2. The method of claim 1, wherein receiving comprises capturing the string of images by controlling a camera of the computing device, and where in the data comprises image data from the camera.

3. The method of claim 2, wherein the software comprises image processing software, wherein controlling the camera further comprises capturing video using the camera, and wherein processing the data further comprises processing the video using the software to detect the string of gestures.

4. The method of claim 2, wherein the camera is a 3-dimensional image capture device, and wherein processing the data further comprises processing three-dimensional images to detect the string of gestures.

5. The method of claim 1, wherein the computing device comprises a smart phone and wherein the software comprises an application executable by the smart phone.

6. The method of claim 5, wherein mapping the string of gestures comprises transmitting information representing the string of gestures to a remote server and wherein identifying words in the written language comprises receiving data representing the words from the remote server.

7. The method of claim 1, further comprising displaying the words to the user via a visual display of the computing device.

8. The method of claim 1, further comprising playing sounds to the user corresponding to mapped phonetic elements.

9. The method of claim 8, further comprising providing the user with an option to selectively enable and disable the playing of the sounds.

10. The method of claim 1, wherein at least one of processing the data and mapping the string of gestures to the string of phonetic elements comprises using a neural net to learn desired user phonetic selections responsive to unique gestures of a particular user.

11. The method of claim 1, wherein each vector comprises a position in n-degree space, and wherein mapping further comprises translating the position in n-degree space to a position in m-degree space, where n>

m, and selecting a phonetic element uniquely associated with the position in m-degree space.

12. The method of claim 11, wherein translating includes accessing a dictionary to map positions in n-degree space to corresponding positions in m-degree space.

13. The method of claim 12, further comprising using principal components analysis to adaptively learn the dictionary.

14. The method of claim 1, wherein the images encompass a hand of the user, wherein the gestures encompass hand poses, and wherein at least some of the vectors correspond to hand poses.

15. The method of claim 1, embodied as a method of inputting words to a computing device.

16. An apparatus comprising instructions stored on non-transitory machine-readable media, the instructions when executed to cause at least one processor to:

receive data representing a string of images, the images representing at least part of a user;

process the data to detect therefrom a string of gestures, each gesture represented by a vector;

map the vectors to a string of phonetic elements; and

automatically identify words in a written language corresponding to a string of the phonetic elements.

17. An apparatus, comprising:

an input device to receive data representing a string of images, the string of images representing at least part of a user; and

at least one processor to

process the data to detect a string of gestures of the user from the images, each gesture corresponding to a vector,

map the vectors to a string of phonetic elements, and

automatically identify words in a written language corresponding to a string of the phonetic elements.

18. The apparatus of claim 17, embodied as a smart phone.

19. The apparatus of claim 17, wherein the input device comprises a 3-dimensional image capture device, and wherein the apparatus further comprises instructions stored on machine-readable media to cause the at least one processor to process three-dimensional images from the 3-dimensional image capture device to detect the string of gestures.

20. The apparatus of claim 17, wherein the at least part of the user includes a hand, wherein each image comprises information sufficient to determine a pose of the hand, and wherein each vector represents a pose of the hand.