GENERATION OF TEXT BY WAY OF A TOUCHLESS INTERFACE

• US 20140368434A1
• Filed: 06/13/2013
• Published: 12/18/2014
• Est. Priority Date: 06/13/2013
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Paek, Timothy S., Apacible, Johnson
• Application Number:
  US13/916,606
• Publication Number:
  US 20140368434A1
• Time in Patent Office: Days
• US Class Current: 345/168
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Described herein are technologies that facilitate decoding a continuous sequence of gestures set forth in the air by a user. A sensor captures movement of a portion of a body of the user relative to a keyboard displayed on a display screen, and a continuous trace is identified based upon the captured movement. The continuous trace is decoded to ascertain a word desirably set forth by the user.

Claims

1. A method, comprising:

receiving data that is indicative of movement of a portion of a body of a user relative to a display screen, the user being displaced from the display screen, the movement of the portion of the body forming a continuous trace;

responsive to receiving the data, identifying the continuous trace;

identifying a word based at least in part upon the continuous trace; and

executing at least one processing operation based at least in part upon the identifying of the word.

2. The method of claim 1, wherein the data that is indicative of movement of the portion of the body of the user relative to the display screen comprises images output by a camera.

3. The method of claim 2, wherein the data that is indicative of the movement of the portion of the body of the user relative to the display screen comprises data output by a depth sensor that is indicative of distance between the user and the displays screen.

4. The method of claim 3, further comprising detecting that the continuous trace has completed based upon the data output by the depth sensor.

5. The method of claim 1, further comprising:

displaying a keyboard on a portion of the display screen, the keyboard comprising a plurality of character keys, each character key in the plurality of character keys being representative of at least one respective character, wherein identifying the word comprises;

detecting that the continuous trace corresponds to the portion of the display screen where the keyboard is displayed;

identifying a first key over which the continuous trace passes; and

identifying a second key over which the continuous trace passes, wherein the word comprises a first character represented by the first key and a second character represented by the second key.

6. The method of claim 5, further comprising displaying graphical data on the display screen that is representative of the continuous trace, wherein the graphical data indicates that the continuous trace passed over the first key and the second key.

7. The method of claim 5, wherein the first key represents a first plurality of characters and the second key represents a second plurality of characters, and identifying the word comprises:

accessing a gesture model responsive to detecting that the continuous trace corresponds to the portion of the display screen where the keyboard is displayed; and

decoding the continuous trace to identify the word based upon the gesture model.

8. The method of claim 1, wherein the portion of the body of the user is an arm of the user.

9. The method of claim 1, wherein the portion of the body of the user is a finger of the user.

10. The method of claim 1, further comprising:

detecting a command that indicates that the continuous trace has been completed; and

identifying the word only after the command has been detected.

11. The method of claim 1, further comprising:

detecting a spoken utterance set forth by the user commensurate in time with continuous trace being identified; and

identifying the word based at least in part upon the spoken utterance set forth by the user and the continuous trace.

12. The method of claim 1, the at least one processing operation comprising transmitting the word to a computing device of another user as at least a portion of a message.

13. A system, comprising:

a processor; and

a memory that comprises a plurality of components that are executed by the processor, the plurality of components comprising;

a receiver component that receives images output by a camera, the images capturing movement of an arm of a user over time relative to a display screen;

a trace identifier component that identifies a continuous trace set forth by the user based upon the movement of the arm captured in the images output by the camera, the continuous trace corresponding to a continuous movement of the arm of the user;

a decoder component that identifies a word based upon the continuous trace identified by the trace identifier component; and

a display component that displays the word decoded by the decoder component.

14. The system of claim 13 comprised by a video game console.

15. The system of claim 13, wherein the receiver component additionally receives depth data output by a depth sensor, the depth data indicative of distance between the arm of the user and the display screen, the trace identifier component identifying the continuous trace based upon the depth data output by the depth sensor.

16. The system of claim 13, wherein the receiver component additionally receives audio data output by a microphone, the audio data comprising a spoken utterance of the user set forth commensurate in time with the continuous trace, the decoder component identifying the word based upon the spoken utterance of the user.

17. The system of claim 13, the plurality of components further comprising a trace identifier component that recognizes a gesture set forth by the user based upon the images output by the camera, wherein the trace identifier component identifies the continuous trace responsive to the trace identifier component recognizing the gesture

18. The system of claim 17, wherein the gesture comprises transition of a hand of the user from an open position to a closed position.

19. The system of claim 13, wherein the display component displays a keyboard on the display screen, the keyboard comprising a plurality of character keys, each character key representative of at least one respective character, the display component further displaying graphical feedback that is indicative of locations of the continuous trace over the keyboard displayed on the display screen.

20. A computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to perform acts comprising:

receiving a first plurality of images of a user from a camera;

receiving, from a depth sensor, first data that is indicative of a distance between the user and a display screen;

detecting an invocation gesture based upon the first plurality of images received from the camera and the first data received from the depth sensor;

responsive to detecting the invocation gesture, displaying a keyboard on the display screen, the keyboard comprising a plurality of character keys, each character key being representative of at least one respective character;

receiving a second plurality of images from the camera;

receiving second data from the depth sensor, the second plurality of images and the second data capturing movement of an arm of the user relative to the keyboard;

identifying a continuous trace based upon the second plurality of images and the second data; and

identifying a word based upon the continuous trace, the word comprising a first character represented by a first character key over which the continuous trace passed and second character represented by a second character key over which the continuous trace passed.