Display method of on-screen keyboard and computer program product and non-transitory computer readable storage medium thereof

• US 10,452,155 B2
• Filed: 04/28/2017
• Issued: 10/22/2019
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Mitake Information Corp.
• Original Assignee: Bing-Yang Yao
• Inventors: Yao, Bing-Yang
• Primary Examiner(s): Abdin, Shaheda A
• Application Number:
  US15/582,189
• Publication Number:
  US 20170315627A1
• Time in Patent Office: 907 Days
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A display method of an on-screen keyboard, applied to a computing device connected to a physical keyboard, a screen and a sensor, the physical keyboard including a plurality of keys, and the sensor being configured to detect and obtain an operating position image, where the display method of the on-screen keyboard includes: displaying an on-screen keyboard at a topmost layer of a displayed image of the screen; reading the operating position image obtained by the sensor; converting the operating position image into an operating position data, where the operating position data includes relative position data of a key which is touched or is approached; and according to the operating position data, displaying a first key mark at a position corresponding to the key which is touched or is approached on the on-screen keyboard.

Claims

1. A display method of an on-screen keyboard, applied to a computing device connected to a physical keyboard, a screen and a sensor, the physical keyboard comprising a plurality of keys, the sensor being configured to detect and obtain an operating position image, and the physical keyboard further responds to a keystroke of the key, so as to output a key scan code to the computing device, wherein the display method of the on-screen keyboard comprises:

displaying an on-screen keyboard at a topmost layer of a displayed image of the screen;

reading the operating position image obtained by the sensor;

converting the operating position image into an operating position data, wherein the operating position data comprises a relative position data of a key which is touched or approached;

according to the operating position data, displaying a first key mark at a position corresponding to the key which is touched or approached on the on-screen keyboard;

reading the key scan code outputted by the physical keyboard and corresponding to the keystroke of the key;

according to the key scan code, displaying a second key mark at a position on the on-screen keyboard which corresponds to the keystroke of the key;

wherein, after displaying the second key mark at the position corresponding to the keystroke of the key on the on-screen keyboard, the display method further comprises;

identifying, according to the key scan code, whether the keystroke of the key is a locking-key type;

determining whether the key scan code corresponding to the key which is released is received; and

if the key scan code corresponding to the key which is released is received and the key is not a locking-key type, cancelling displaying of the corresponding second key mark; and

further comprising;

an automatic correction mechanism which, when the second key mark is individually marked on the on-screen keyboard, calculates a difference between an erroneously determined position and an actual position, and feeds back the difference to the computing device, so as to correct a parameter for generating the operating position data at a next time.

2. The display method according to claim 1, wherein the sensor is a camera.

3. The display method according to claim 1, after displaying the first key mark at the position corresponding to the key which is touched or is approached on the on-screen keyboard, further comprising:

if the operating position data comprising the relative position data of the key which is touched or is approached is not received again within a preset time, canceling displaying of the corresponding first key mark.

4. The display method according to claim 1, further comprising:

hiding the on-screen keyboard when the operating position data comprising the relative position data of the key which is touched or is approached and the key scan code are not detected after an idle timeout.

5. The display method according to claim 1, wherein shapes of the first key mark and the second key mark are different, the colors of the first key mark and the second key mark are different, or shapes and colors of the first key mark and the second key mark are different.

6. The display method according to claim 1, wherein the first key mark or the second key mark has incomplete transparency.

7. The display method according to claim 1, wherein the on-screen keyboard has incomplete transparency.

8. A computer program product, comprising a plurality of computer executable instructions stored in a non-transitory computer readable medium, wherein the computer executable instructions are loaded and executed by a computing device connected to a physical keyboard, a screen and a sensor to cause the computing device to implement a display method of an on-screen keyboard, the physical keyboard comprising a plurality of keys, the sensor being configured to detect and obtain an operating position image, and the physical keyboard further responds to a keystroke of the key, so as to output a key scan code to the computing device wherein the display method of the on-screen keyboard comprises:

displaying an on-screen keyboard at a topmost layer of a displayed image of the screen;

reading the operating position image obtained by the sensor;

converting the operating position image into an operating position data, wherein the operating position data comprises a relative position data of a key which is touched or approached;

according to the operating position data, displaying a first key mark at a position corresponding to the key which is touched or approached on the on-screen keyboard;

reading the key scan code outputted by the physical keyboard and corresponding to the keystroke of the key; and

according to the key scan code, displaying a second key mark at a position on the on-screen keyboard which corresponds to the keystroke of the key;

wherein, after displaying the second key mark at the position corresponding to the keystroke of the key on the on-screen keyboard, the display method further comprises;

identifying, according to the key scan code, whether the keystroke of the key is a locking-key type;

determining whether the key scan code corresponding to the key which is released is received; and

if the key scan code corresponding to the key which is released is received and the key is not a locking-key type, cancelling displaying of the corresponding second key mark; and

further comprising;

an automatic correction mechanism which, when the second key mark is individually marked on the on-screen keyboard, calculates a difference between an erroneously determined position and an actual position, and feeds back the difference to the computing device, so as to correct a parameter for generating the operating position data at a next time.

9. The computer program product for completing the display method of an on-screen keyboard according to claim 8, wherein the sensor is a camera.

10. The computer program product for completing the display method of an on-screen keyboard according to claim 8, after displaying the first key mark at the position corresponding to the key which is touched or is approached on the on-screen keyboard, further comprising:

if the operating position data comprising the relative position data of the key which is touched or is approached is not received again within a preset time, canceling displaying of the corresponding first key mark.

11. The computer program product for completing the display method of an on-screen keyboard according to claim 8, after displaying the second key mark at the position corresponding to the keystroke of the key on the on-screen keyboard, further comprising:

identifying, according to the key scan code, whether the keystroke of the key is a locking-key type;

determining whether the key scan code corresponding to the key which is released is received; and

if the key scan code corresponding to the key which is released is received and the key is not a locking-key type, cancelling displaying of the corresponding second key mark.

12. The computer program product for completing the display method of an on-screen keyboard according to claim 8, further comprising:

hiding the on-screen keyboard when the operating position data comprising the relative position data of the key which is touched or is approached and the key scan code are not detected after an idle timeout.

13. The computer program product for completing the display method of an on-screen keyboard according to claim 8, wherein shapes of the first key mark and the second key mark are different, the colors of the first key mark and the second key mark are different, or shapes and colors of the first key mark and the second key mark are different.

14. A non-transitory computer readable storage medium, storing a computer program comprising a plurality of computer executable instructions that, when executed by a computing device connected to a physical keyboard, a screen and a sensor, cause the computing device to implement a display method of an on-screen keyboard, the physical keyboard comprising a plurality of keys, and the sensor being configured to detect and obtain an operating position image, and the physical keyboard further responds to a keystroke of the key, so as to output a key scan code to the computing device, wherein the display method of the on-screen keyboard comprises:

displaying an on-screen keyboard at a topmost layer of a displayed image of the screen;

reading the operating position image obtained by the sensor;

converting the operating position image into an operating position data, wherein the operating position data comprises a relative position data of a key which is touched or approached;

according to the operating position data, displaying a first key mark at a position corresponding to the key which is touched or approached on the on-screen keyboard;

after displaying the first key mark at the position corresponding to the key which is touched or is approached on the on-screen keyboard, if the operating position data comprising the relative position data of the key which is touched or is approached is not received again within a preset time, canceling displaying of the corresponding first key mark; and

reading the key scan code outputted by the physical keyboard and corresponding to the keystroke of the key;

according to the key scan code, displaying a second key mark at a position on the on-screen keyboard which corresponds to the keystroke of the key; and

after displaying the second key mark at the position corresponding to the keystroke of the key on the on-screen keyboard, further comprising;

identifying, according to the key scan code, whether the keystroke of the key is a locking-key type;

determining whether the key scan code corresponding to the key which is released is received; and

if the key scan code corresponding to the key which is released is received and the key is not a locking-key type, cancelling displaying of the corresponding second key mark.

15. The non-transitory computer readable storage medium according to claim 14, wherein shapes of the first key mark and second key mark are different, the colors of the first key mark and the second key mark are different, or shapes and colors of the first key mark and the second key mark are different.