System and method for receiving user input and program storage medium thereof

• US 9,857,971 B2
• Filed: 06/12/2014
• Issued: 01/02/2018
• Est. Priority Date: 12/02/2013
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Chang, Ming-Wei, Chang, Chia-Ming, Chiueh, Tzi-Cker
• Primary Examiner(s): Nabi, Reza
• Assistant Examiner(s): Rodriguez, Daniel
• Application Number:
  US14/303,438
• Publication Number:
  US 20150153950A1
• Time in Patent Office: 1,300 Days
• Field of Search: 715773
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06V 40/113 Recognition of static hand signs

Abstract

A user input method includes the following steps. A virtual keyboard layout and a control region are displayed. The virtual keyboard layout includes a plurality of key subgroups each mapped to a respective one of a plurality of regions of the control region. Locations of an object from at least one captured image are extracted to identify a location of a feature point of the object. A target region in which the feature point is located is determined. Keys mapped to the target region are determined. Movements of the object are translated as input data to the user interface system.

Claims

1. A method for receiving input data comprising:

displaying a virtual keyboard layout and a control region by a computing system, the virtual keyboard layout including a plurality of keys, and the plurality of keys being divided into a plurality of key subgroups each mapped to a respective one of a plurality of regions of the control region, including determining to display the virtual keyboard layout and the control region free of correspondence to a user hand;

extracting, by the computing system, a plurality of locations of points of the user hand from at least one captured image to identify a location of a feature point of the user hand;

comparing, by the computing system, the location of the feature point of the user hand with the plurality of regions of the control region to determine, among the plurality of regions of the control region, a target region in which the feature point of the user hand is located;

highlighting a plurality of mapped-keys from the plurality of keys according to the target region;

after the target region is determined, displaying a first active identifier surrounding the plurality of mapped-keys;

after a location of a user finger of the user hand is detected, displaying, near the location of said user finger, a first group of keys that include same keys as the plurality of mapped-keys;

in response to an action of said user finger, displaying a second active identifier surrounding one key of the first group of keys; and

determining one of the mapped-keys as input data to a user interface system according to the action of said user finger.

2. The method as claimed in claim 1, wherein a respective key subgroup of the virtual keyboard layout is mapped to the respective one of the plurality of regions of the control region based on a position of the plurality of key subgroups in the virtual keyboard layout and a position of the respective one of the plurality of regions in the control region.

3. The method as claimed in claim 1, wherein the step of extracting the plurality of locations of the points of the user hand includes:

extracting at least one image of at least one of a plurality of user fingers, a plurality of user fingertips and/or the feature point of at least one user hand from the at least one captured image.

4. The method as claimed in claim 1, wherein

the feature point includes at least one point of a back of at least one user hand, and

the at least one point of the back of the at least one user hand includes a thumb-index web or a center of the back of the at least one user hand.

5. The method as claimed in claim 1, further comprising placing a glove or gloves having a feature point marker and a plurality of fingernail-type markers on the user hand, wherein the extracting of the plurality of locations of the points of the user hand includes:

extracting a plurality of images of the plurality of fingernail-type markers and the feature point marker from the at least one captured image.

8. The method as claimed in claim 1, wherein the action of said user finger includes moving said user finger, tapping said user finger, bending said user finger, or clicking by said user finger.

6. The method as claimed in 1, further comprising a step of:

displaying at least one of an input result, tracking information and the at least one captured image of the user hand.

7. The method as claimed in 1, wherein the plurality of regions of the control region are combined into the plurality of key subgroups of the virtual keyboard layout, and a plurality of locations of the plurality of regions of the control region are overlapped with the location of the plurality of key subgroups of the virtual keyboard layout.

9. A non-transitory program storage medium storing a computer program for causing an electronic device to perform the steps of:

displaying a virtual keyboard layout and a control region, the virtual keyboard layout including a plurality of keys, and the plurality of keys being divided into a plurality of key subgroups each mapped to a respective one of a plurality of regions of the control region, including determining to display the virtual keyboard layout and the control region free of correspondence to a user hand;

extracting a plurality of locations of points of the user hand from at least one captured image to identify a location of a feature point of the user hand;

comparing the location of the feature point of the object with the plurality of regions of the control region to determine, among the plurality of regions of the control region, a target region in which the feature point of the user hand is located;

highlighting a plurality of mapped-keys from the plurality of keys according to the target region;

after the target region is determined, displaying a first active identifier surrounding the plurality of mapped-keys;

after a location of a user finger of the user hand is detected, displaying, near the location of said user finger, a first group of keys which include same keys as the plurality of mapped-keys;

in response to an action of said user finger, displaying a second active identifier surrounding one key of the first group of keys; and

determining one of the mapped-keys as input data to a user interface system according to the action of said user finger.

10. The program storage medium as claimed in claim 9, wherein a respective key subgroup of the virtual keyboard layout is mapped to the respective one of the plurality of regions of the control region based on a position of the plurality of key subgroups in the virtual keyboard layout and a position of the respective one of the plurality of regions in the control region.

11. The program storage medium as claimed in claim 9, wherein the step of extracting the plurality of locations of the points of the user hand includes:

extracting at least one image of at least one of a plurality of user fingers, a plurality of user fingertips and/or the feature point of at least one user hand from the at least one captured image.

12. The program storage medium as claimed in claim 9, wherein

the feature point includes at least one point of a back of at least one user hand, and

the at least one point of the back of the at least one user hand includes a thumb-index web or a center of the back of the at least one user hand.

13. The program storage medium according to claim 9, wherein

the user hand has thereon a glove or gloves having a feature point marker and a plurality of fingernail-type markers;

and

the steps of extracting the plurality of locations of the points of the user hand includes;

extracting images of the plurality of fingernail-type markers and the feature point marker from the at least one captured image.

14. The program storage medium as claimed in claim 9, wherein the program storage medium causes the electronic device to further perform a step of:

displaying at least one of an input result, tracking information and the at least one captured image of the user hand.

15. The program storage medium as claimed in claim 9, wherein the plurality of regions of the control region are combined into the plurality of key subgroups of the virtual keyboard layout, and a plurality of locations of the plurality of regions of the control region are overlapped with a plurality of locations of the plurality of key subgroups of the virtual keyboard layout.

16. The program storage medium as claimed in claim 9, wherein the action of said user finger includes moving said user finger, tapping said user finger, bending said user finger, or clicking by said user finger.