VIRTUAL KEYBOARDS AND METHODS OF PROVIDING THE SAME

• US 20120268376A1
• Filed: 04/20/2011
• Published: 10/25/2012
• Est. Priority Date: 04/20/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Bi, Ning
• Granted Patent: US 8,928,589 B2
• Time in Patent Office: Days
• US Class Current: 345/168
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

The present disclosure provides systems, methods and apparatus, including computer programs encoded on computer storage media, for providing virtual keyboards. In one aspect, a system includes a camera, a display, a video feature extraction module and a gesture pattern matching module. The camera captures a sequence of images containing a finger of a user, and the display displays each image combined with a virtual keyboard having a plurality of virtual keys. The video feature extraction module detects motion of the finger in the sequence of images relative to virtual sensors of the virtual keys, and determines sensor actuation data based on the detected motion relative to the virtual sensors. The gesture pattern matching module uses the sensor actuation data to recognize a gesture.

Claims

1. A system for inputting data into a computing device, the system comprising:

a camera for capturing a sequence of images containing a finger of a user;

a display for displaying each image of the sequence combined with a virtual keyboard, wherein the position of the finger in the displayed image relative to the virtual keyboard changes as the finger of the user moves relative to the camera;

a video feature extraction module configured to detect motion of the user'"'"'s finger in the sequence of images relative to the virtual keyboard, wherein the virtual keyboard comprises a plurality of virtual keys, each virtual key having a plurality of virtual sensors, and wherein the video feature extraction module captures sensor actuation data of the virtual sensors using the detected motion; and

a gesture pattern matching module for using the sensor actuation data to recognize a user'"'"'s gesture and input data into the computing device.

2. The system of claim 1, wherein the display overlays the virtual keyboard on each of the images in the sequence, the virtual keyboard being at least partially transparent.

3. The system of claim 1, wherein the system is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the gesture is recognized.

4. The system of claim 1, further comprising a gesture endpoint detection module for detecting a gesture endpoint.

5. The system of claim 4, wherein the gesture pattern matching module is configured to compare the sensor actuation data to each of a plurality of gesture recognition templates to recognize the gesture.

6. The system of claim 4, wherein the gesture pattern matching module is configured to compare the sensor actuation data to each of the plurality of gesture recognition templates after the gesture endpoint is detected.

7. The system of claim 5, wherein the gesture pattern matching module is configured to align the sensor actuation data to each of the plurality of gesture recognition templates using dynamic time warping.

8. The system of claim 1, further comprising a false alarm rejection module for detecting a motion condition using the sensor actuation data, and to prevent the gesture pattern matching module from recognizing the gesture when the motion condition is above a threshold.

9. The system of claim 8, wherein the false alarm rejection module is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the motion condition is above the threshold.

10. The system of claim 1, wherein the gesture is a key-stroke.

11. A method for inputting data into an electronic device using a virtual keyboard, the method comprising:

receiving a sequence of images containing a finger of a user;

generating a combined image for each image of the sequence, the combined image containing the finger of the user and a virtual keyboard;

detecting a motion of the finger in the sequence of images relative to the virtual keyboard, the virtual keyboard having a plurality of virtual keys, each virtual key having a plurality of virtual sensors;

generating sensor actuation data by comparing a pixel region of the detected motion to a pixel region of the virtual sensors; and

recognizing a gesture using the sensor actuation data to input data into the electronic device.

12. The method of claim 11, further comprising displaying the combined image on a display.

13. The method of claim 12, wherein displaying the combined image comprises displaying the virtual keyboard overlaid on the sequence of images, the virtual keyboard being at least partially transparent.

14. The method of claim 12, further comprising highlighting at least one virtual key after recognizing the gesture.

15. The method of claim 11, wherein recognizing the gesture comprises comparing the sensor actuation data to each of a plurality of gesture recognition templates to identify at least one candidate gesture.

16. The method of claim 15, further comprising detecting a gesture endpoint using the detected motion.

17. The method of claim 16, further comprising aligning the sensor actuation data with each gesture recognition template using dynamic time warping.

18. The method of claim 15, further comprising assigning a confidence factor to each candidate gesture.

19. The method of claim 18, wherein recognizing the gesture comprises selecting a gesture from the at least one candidate gesture having the greatest confidence factor.

20. The method of claim 15, further comprising determining a motion condition using the sensor actuation data, and preventing one or more of the at least one candidate gestures from being the recognized gesture when the motion condition is above a threshold.

21. A system for inputting data into a computing device, the system comprising:

means for capturing a sequence of images containing a finger of a user;

means for displaying each image of the sequence combined with a virtual keyboard, wherein the position of the finger in the displayed image relative to the virtual keyboard changes as the finger of the user moves relative to the camera;

means for detecting motion of the user'"'"'s finger in the sequence of images relative to the virtual keyboard, wherein the virtual keyboard comprises a plurality of virtual keys, each virtual key having a plurality of virtual sensors, and wherein means for detecting motion captures sensor actuation data of the virtual sensors using the detected motion; and

means for recognizing a user'"'"'s gesture using the sensor actuation data to input data into the computing device.

22. The system of claim 21, wherein the means for displaying overlays the virtual keyboard on each of the images in the sequence, the virtual keyboard being at least partially transparent.

23. The system of claim 21, wherein the system is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the gesture is recognized.

24. The system of claim 21, further comprising a means for detecting a gesture endpoint.

25. The system of claim 24, wherein the means for recognizing the user'"'"'s gesture is configured to compare the sensor actuation data to each of a plurality of gesture recognition templates to recognize the gesture.

26. The system of claim 25, wherein the means for recognizing the user'"'"'s gesture is configured to compare the sensor actuation data to each of the plurality of gesture recognition templates after the gesture endpoint is detected.

27. The system of claim 25, wherein means for recognizing the user'"'"'s gesture is configured to align the sensor actuation data to each of the plurality of gesture recognition templates using dynamic time warping.

28. The system of claim 21, further comprising a means for preventing the gesture pattern matching module from recognizing the gesture when the motion condition is above a threshold.

29. A non-transitory computer-readable storage medium comprising instructions that when executed perform a method of inputting data into an electronic device using a virtual keyboard, the method comprising:

receiving a sequence of images containing a finger of a user;

generating a combined image for each image of the sequence, the combined image containing the finger of the user and a virtual keyboard;

detecting a motion of the finger in the sequence of images relative to the virtual keyboard, the virtual keyboard having a plurality of virtual keys, each virtual key having a plurality of virtual sensors;

generating sensor actuation data by comparing a pixel region of the detected motion to a pixel region of the virtual sensors; and

recognizing a gesture using the sensor actuation data to input data into the electronic device.

30. The computer-readable storage medium of claim 29, wherein the method further comprises displaying the combined image on a display.

31. The computer-readable storage medium of claim 30, wherein displaying the combined image comprises displaying the virtual keyboard overlaid on the sequence of images, the virtual keyboard being at least partially transparent.

32. The computer-readable storage medium of claim 31, wherein the method further comprises highlighting at least one virtual key after recognizing the gesture.

33. The computer-readable storage medium of claim 29, wherein recognizing the gesture comprises comparing the sensor actuation data to each of a plurality of gesture recognition templates to identify at least one candidate gesture.

34. The computer-readable storage medium of claim 33, wherein the method further comprises detecting a gesture endpoint using the detected motion.

35. The computer-readable storage medium of claim 34, wherein the method further comprises aligning the sensor actuation data with each gesture recognition template using dynamic time warping.

36. The computer-readable storage medium of claim 35, wherein the method further comprises assigning a confidence factor to each candidate gesture.

37. The computer-readable storage medium of claim 36, wherein recognizing the gesture comprises selecting a gesture from the at least one candidate gesture having the greatest confidence factor.

38. The computer-readable storage medium of claim 37, wherein the method further comprises determining a motion condition using the sensor actuation data, and preventing one or more of the at least one candidate gestures from being the recognized gesture when the motion condition is above a threshold.