Virtual keyboard for a non-tactile three dimensional user interface

• US 8,959,013 B2
• Filed: 09/25/2011
• Issued: 02/17/2015
• Est. Priority Date: 09/27/2010
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Galor, Micha, Or, Ofir, Litvak, Shai, Sali, Erez
• Primary Examiner(s): Shah, Paras D
• Assistant Examiner(s): THOMAS-HOMESCU, ANNE L
• Application Number:
  US13/244,490
• Publication Number:
  US 20120078614A1
• Time in Patent Office: 1,241 Days
• Field of Search: 345/168, 345/156, 345/173, 345/441, 345/177, 345/158, 345/174, 345/169, 382/154, 382/106, 382/190, 382/103, 715/863, 715/702, 715/773, 704 2- 10
• US Class Current: 704/9
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 40/274 Converting codes to words; Guess-ahead of partial word inputs

Abstract

A method, including presenting, by a computer system executing a non-tactile three dimensional user interface, a virtual keyboard on a display, the virtual keyboard including multiple virtual keys, and capturing a sequence of depth maps over time of a body part of a human subject. On the display, a cursor is presented at positions indicated by the body part in the captured sequence of depth maps, and one of the multiple virtual keys is selected in response to an interruption of a motion of the presented cursor in proximity to the one of the multiple virtual keys.

Claims

1. A method, comprising:

presenting, by a computer system executing a non-tactile three dimensional user interface, a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys;

capturing a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane;

presenting, on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane; and

selecting one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor'"'"'s trajectory path segment over the virtual keys, wherein the trajectory of the user'"'"'s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and

wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs.

2. The method according to claim 1, wherein selecting one of the multiple virtual keys comprises using a language model.

3. The method according to claim 2, wherein the language model is selected from a list consisting of a dictionary, a statistical dictionary, an n-gram model, a Markov model and a dynamic Bayesian network.

4. The method according to claim 2, wherein the language model applies rules specific to a given language.

5. The method according to claim 4, wherein the rules are selected from a list consisting of word rules, short phrase rules, parts of speech rules and grammatical rules.

6. The method according to claim 2, wherein the language model utilizes a custom dictionary based on text previously entered by a user interacting with the non-tactile three dimensional user interface.

7. The method according to claim 2, wherein the language model utilizes a custom dictionary specific to an application executing on the computer system.

8. The method according to claim 2, and comprising the language model selecting none of the virtual keys if none of the virtual keys that are in proximity to the cursor are sufficiently probable.

9. The method according to claim 1, and comprising presenting visual feedback of the selected one of the multiple virtual keys on the display.

10. The method according to claim 1, wherein each of the multiple virtual keys is selected from a list consisting of alphanumeric characters, symbol characters, punctuation characters and control commands.

11. The method according to claim 1, wherein presenting the cursor in proximity to the one of the multiple virtual keys comprises presenting the cursor within a border surrounding the one of the multiple virtual keys.

12. The method according to claim 1, and comprising conveying visual feedback upon presenting the cursor in proximity to the one of the multiple virtual keys.

13. The method according to claim 1, and comprising conveying visual feedback when a user engages and disengages from the virtual keyboard.

14. An apparatus, comprising:

a display; and

a computer executing a non-tactile three dimensional user interface and configured to present a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys, to capture a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane, to present, on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane, and to select one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor'"'"'s trajectory path segment over the virtual keys, wherein the trajectory of the user'"'"'s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and

wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs.

15. The apparatus according to claim 14, wherein the computer is configured to select one of the multiple virtual keys by using a language model.

16. The apparatus according to claim 15, wherein the computer is configured to select the language model from a list consisting of a dictionary, a statistical dictionary, an n-gram model, a Markov model and a dynamic Bayesian network.

17. The apparatus according to claim 15, wherein the language model applies rules specific to a given language.

18. The apparatus according to claim 17, wherein the computer is configured to select the rules from a list consisting of word rules, short phrase rules, parts of speech rules and grammatical rules.

19. The apparatus according to claim 15, wherein the language model utilizes a custom dictionary based on text previously entered by a user interacting with the non-tactile three dimensional user interface.

20. The apparatus according to claim 15, wherein the language model utilizes a custom dictionary specific to an application executing on the computer system.

21. The apparatus according to claim 15, wherein the computer is configured to select none of the virtual keys if none of the virtual keys that are in proximity to the cursor are sufficiently probable.

22. The apparatus according to claim 14, wherein the computer is configured to present visual feedback of the selected one of the multiple virtual keys on the display.

23. The apparatus according to claim 14, wherein the computer is configured to select each of the multiple virtual keys from a list consisting of alphanumeric characters, symbol characters, punctuation characters and control commands.

24. The apparatus according to claim 14, wherein the computer is configured to present the cursor in proximity to the one of the multiple virtual keys by presenting the cursor within a border surrounding the one of the multiple virtual keys.

25. The apparatus according to claim 14, wherein the computer is configured to conveying visual feedback upon presenting the cursor in proximity to the one of the multiple virtual keys.

26. The apparatus according to claim 14, wherein the computer is configured to conveying visual feedback when a user engages and disengages from the virtual keyboard.

27. A computer software product comprising a non-transitory computer-readable medium, in which program instructions are stored, which instructions, when read by a computer executing a non-tactile three dimensional user interface, cause the computer:

to present a virtual keyboard on a display, the virtual keyboard comprising multiple virtual keys, to capture a sequence of depth maps, via a 3D capturing device, over time of a hand of a human subject while the human subject moves the hand in a plane, to present on the display, a cursor at positions indicated by the hand in the captured sequence of depth maps such that the cursor moves over the virtual keys on the display in response to movement of the hand in the plane, and to select one of the multiple virtual keys if a change in direction of trajectory is determined by using the captured sequence of depth maps and calculating points along the presented cursor'"'"'s trajectory path segment over the virtual keys, wherein the trajectory of the user'"'"'s hand tracks a plurality of keys for which a cursor crosses spatially from a previously determined key point to a newly desired key point, and wherein the keys between the previously determined key point to the newly desired key point are used in combination with previously determined key inputs to configure a language model for estimating probable words and estimating a most likely key from the keys based on the language model and displaying the most likely key appended with the previously determined key inputs; and

wherein if the change of trajectory is not detected then determining whether the cursor is in proximity to a plurality of keys for a standard time period and selecting keys to be used by the language model for estimating a most likely key from the plurality of keys and displaying the most likely key appended with the previously determined key inputs.