Virtual keyboards and methods of providing the same

• US 8,928,589 B2
• Filed: 04/20/2011
• Issued: 01/06/2015
• Est. Priority Date: 04/20/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Bi, Ning
• Primary Examiner(s): Mengistu, Amare
• Assistant Examiner(s): Jansen, II, Michael J
• Application Number:
  US13/090,497
• Publication Number:
  US 20120268376A1
• Time in Patent Office: 1,357 Days
• Field of Search: 345/168, 345/169, 345/175, 715/757, 715/771, 715/773
• US Class Current: 345/168
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886

Abstract

The present disclosure provides systems, methods and apparatus, including computer programs encoded on computer storage media, for providing virtual keyboards. In one aspect, a system includes a camera, a display, a video feature extraction module and a gesture pattern matching module. The camera captures a sequence of images containing a finger of a user, and the display displays each image combined with a virtual keyboard having a plurality of virtual keys. The video feature extraction module detects motion of the finger in the sequence of images relative to virtual sensors of the virtual keys, and determines sensor actuation data based on the detected motion relative to the virtual sensors. The gesture pattern matching module uses the sensor actuation data to recognize a gesture.

Claims

1. A system for inputting data into a computing device, the system comprising:

a camera for capturing a sequence of images containing a finger of a user;

a virtual keyboard comprising a plurality of virtual keys, each virtual key having a plurality of virtual sensors, each virtual sensor comprising a region of adjoining pixels within a corresponding virtual key, wherein a plurality of the virtual keys have different virtual sensor layouts from one another;

a display for displaying each image of the sequence combined with the virtual keyboard, wherein the position of the finger in the displayed image relative to the virtual keyboard changes as the finger of the user moves relative to the camera;

a video feature extraction module configured to;

detect motion of the user'"'"'s finger in the sequence of images relative to the virtual keyboard,

identify a sequence of actuated virtual sensors within at least one of the plurality of virtual keys based on the detected motion, wherein a virtual sensor is actuated when the motion of the user'"'"'s finger is detected over a threshold number of pixels within the region of adjoining pixels corresponding to the actuated virtual sensor, and

collect virtual sensor actuation data comprising both geographical and temporal information related to each actuated virtual sensor of the sequence of actuated virtual sensors; and

a gesture pattern matching module for using the sequence of virtual sensor actuations to recognize a user'"'"'s gesture as one of a valid keystroke and a rejected keystroke and input data into the computing device.

2. The system of claim 1, wherein the display overlays the virtual keyboard on each of the images in the sequence, the virtual keyboard being at least partially transparent.

3. The system of claim 1, wherein the system is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the gesture is recognized.

4. The system of claim 1, further comprising a gesture endpoint detection module for detecting a gesture endpoint.

5. The system of claim 4, wherein the gesture pattern matching module is configured to compare the sensor actuation data to each of a plurality of gesture recognition templates to recognize the gesture.

6. The system of claim 4, wherein the gesture pattern matching module is configured to compare the sensor actuation data to each of the plurality of gesture recognition templates after the gesture endpoint is detected.

7. The system of claim 5, wherein the gesture pattern matching module is configured to align the senor actuation data to each of the plurality of gesture recognition templates using dynamic time warping.

8. The system of claim 1, further comprising a false alarm rejection module for detecting a motion condition using the sensor actuation data, and to prevent the gesture pattern matching module from recognizing the gesture when the motion condition is above a threshold.

9. The system of claim 8, wherein the false alarm rejection module is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the motion condition is above the threshold.

38. The system of claim 1, wherein motion of the user'"'"'s finger is translated into a motion vector having a magnitude.

39. The system of claim 38, wherein the at least one actuated virtual sensor is actuated when the magnitude of the motion vector is above a threshold.

40. The system of claim 39, wherein a virtual sensor is not actuated when the user'"'"'s finger is in a substantially stationary position over the virtual sensor.

10. A method for inputting data into an electronic device using a virtual keyboard, the method comprising:

receiving a sequence of images containing a finger of a user;

generating a combined image for each image of the sequence, the combined image containing the finger of the user and a virtual keyboard, the virtual keyboard comprising a plurality of virtual keys, each virtual key having a plurality of virtual sensors, each virtual sensor comprising a region of adjoining pixels within a corresponding virtual key, wherein a plurality of the virtual keys have different virtual sensor layouts from one another;

detecting a motion of the finger in the sequence of images relative to the virtual sensors;

identifying a sequence of actuated virtual sensors within at least one of the plurality of virtual keys based on the detected motion, wherein a virtual sensor is actuated when the motion of the user'"'"'s finger is detected over a threshold number of pixels within the plurality of pixels corresponding to the actuated virtual sensor;

generating sensor actuation data comprising both geographical and temporal information related to each actuated virtual sensor of the sequence of actuated virtual sensors; and

recognizing a gesture as one of a valid keystroke and a rejected keystroke using the sequence of virtual sensor actuations to input data into the electronic device.

11. The method of claim 10, further comprising displaying the combined image on a display.

12. The method of claim 11, wherein displaying the combined image comprises displaying the virtual keyboard overlaid on the sequence of images, the virtual keyboard being at least partially transparent.

13. The method of claim 11, further comprising highlighting at least one virtual key after recognizing the gesture.

14. The method of claim 10, wherein recognizing the gesture comprises comparing the senor actuation data to each of a plurality of gesture recognition templates to identify at least one candidate gesture.

15. The method of claim 14, further comprising detecting a gesture endpoint using the detected motion.

16. The method of claim 15, further comprising aligning the sensor actuation data with each gesture recognition template using dynamic time warping.

17. The method of claim 14, further comprising assigning a confidence factor to each candidate gesture.

18. The method of claim 17, wherein recognizing the gesture comprises selecting a gesture from the at least one candidate gesture having the greatest confidence factor.

19. The method of claim 14, further comprising determining a motion condition using the sensor actuation data, and preventing one or more of the at least one candidate gestures from being the recognized gesture when the motion condition is above a threshold.

20. A system for inputting data into a computing device, the system comprising:

means for capturing a sequence of images containing a finger of a user;

means for displaying each image of the sequence combined with a virtual keyboard, the virtual keyboard comprising a plurality of virtual keys, each virtual key having a plurality of virtual sensors, each virtual sensor comprising a region of adjoining pixels within a corresponding virtual key, wherein a plurality of the virtual keys have different virtual sensor layouts from one another, wherein the position of the finger in the displayed image relative to the virtual keyboard changes as the finger of the user moves relative to the camera;

means for detecting motion of the user'"'"'s finger in the sequence of images relative to the virtual sensors;

means for identifying a sequence of actuated virtual sensors within at least one of the plurality of virtual keys based on the detected motion, wherein a virtual sensor is actuated when the motion of the user'"'"'s finger is detected over a threshold number of pixels within the plurality of pixels corresponding to the actuated virtual sensor;

means for generating sensor actuation data comprising both geographical and temporal information related to each actuated virtual sensor of the sequence of actuated virtual sensors; and

means for recognizing a user'"'"'s gesture as one of a valid keystroke and a rejected keystroke using the sequence of virtual sensor actuations data to input data into the computing device.

21. The system of claim 20, wherein the means for displaying overlays the virtual keyboard on each of the images in the sequence, the virtual keyboard being at least partially transparent.

22. The system of claim 20, wherein the system is configured to provide at least one of visual feedback, audio feedback, and vibratory feedback when the gesture is recognized.

23. The system of claim 20, further comprising a means for detecting a gesture endpoint.

24. The system of claim 23, wherein the means for recognizing the user'"'"'s gesture is configured to compare the sensor actuation data to each of a plurality of gesture recognition templates to recognize the gesture.

25. The system of claim 24, wherein the means for recognizing the user'"'"'s gesture is configured to compare the sensor actuation data to each of the plurality of gesture recognition templates after the gesture endpoint is detected.

26. The system of claim 24, wherein means for recognizing the user'"'"'s gesture is configured to align the senor actuation data to each of the plurality of gesture recognition templates using dynamic time warping.

27. The system of claim 20, further comprising a means for preventing the gesture pattern matching module from recognizing the gesture when the motion condition is above a threshold.

28. A non-transitory computer-readable storage medium comprising instructions that when executed perform a method of inputting data into an electronic device using a virtual keyboard, the method comprising:

receiving a sequence of images containing a finger of a user;

generating a combined image for each image of the sequence, the combined image containing the finger of the user and a virtual keyboard, the virtual keyboard comprising a plurality of virtual keys, each virtual key having a plurality of virtual sensors, each virtual sensor comprising a region of adjoining pixels within a corresponding virtual key, wherein a plurality of the virtual keys have different virtual sensor layouts from one another;

detecting a motion of the finger in the sequence of images relative to the virtual sensors;

identifying a sequence of actuated virtual sensors within at least one of the plurality of virtual keys based on the detected motion, wherein a virtual sensor is actuated when the motion of the user'"'"'s finger is detected over a threshold number of pixels within the plurality of pixels corresponding to the actuated virtual sensor;

generating sensor actuation data comprising both geographical and temporal information related to each actuated virtual sensor of the sequence of actuated virtual sensors; and

recognizing a gesture as one of a valid keystroke and a rejected keystroke using the sequence of virtual sensor actuations to input data into the electronic device.

29. The computer-readable storage medium of claim 28, wherein the method further comprises displaying the combined image on a display.

30. The computer-readable storage medium of claim 29, wherein displaying the combined image comprises displaying the virtual keyboard overlaid on the sequence of images, the virtual keyboard being at least partially transparent.

31. The computer-readable storage medium of claim 30, wherein the method further comprises highlighting at least one virtual key after recognizing the gesture.

32. The computer-readable storage medium of claim 28, wherein recognizing the gesture comprises comparing the senor actuation data to each of a plurality of gesture recognition templates to identify at least one candidate gesture.

33. The computer-readable storage medium of claim 32, wherein the method further comprises detecting a gesture endpoint using the detected motion.

34. The computer-readable storage medium of claim 33, wherein the method further comprises aligning the sensor actuation data with each gesture recognition template using dynamic time warping.

35. The computer-readable storage medium of claim 34, wherein the method further comprises assigning a confidence factor to each candidate gesture.

36. The computer-readable storage medium of claim 35, wherein recognizing the gesture comprises selecting a gesture from the at least one candidate gesture having the greatest confidence factor.

37. The computer-readable storage medium of claim 36, wherein the method further comprises determining a motion condition using the sensor actuation data, and preventing one or more of the at least one candidate gestures from being the recognized gesture when the motion condition is above a threshold.