Method and device for optical input of commands or data

• US 5,767,842 A
• Filed: 04/21/1995
• Issued: 06/16/1998
• Est. Priority Date: 02/07/1992
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Korth, Hans-Erdmann
• Primary Examiner(s): WU, XIAO MIN
• Application Number:
  US08/427,188
• Time in Patent Office: 1,152 Days
• Field of Search: 345/157, 345/158, 345/163, 345/168, 345/169, 345/173, 382/312, 382/313, 382/315
• US Class Current: 345/168
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: H01H 2217/044 Repetitive strain injury [RSI] considerations

Abstract

A method of user command or data acquisition in a processing system that allows for analysis of human hand and finger motions. The activities are then interpreted as operations on a physically non-existent input device. The resulting optical user/computer interface complements the visual screen-to-user path with a hand-to-image data acquisition link from the user to the processor unit. It allows for the emulation of different types of manually operated input devices (mouse, keyboard, etc.). Mechanical input units can be replaced by virtual devices, optimized for the current application and for the user'"'"'s physiology. The speed, simplicity and avoidance of ambiguity of manual data input will be maintained. Sensing of finger motions can generate feedback to reduce the hazard of repetitive strain injury (RSI) syndrome.

Claims

1. In a data processing system, a method of data input using video sensors for monitoring positions of an operator'"'"'s hand with respect to keys on a virtual keyboard optically produced on a surface and for subsequent and correlated data recognition by corresponding key locations on the keyboard to the monitored positions of the fingers of the operator'"'"'s hand, comprising the steps of:

touching the keys with fingers of the operator'"'"'s hand to input data;

scanning by the video sensors adjacent to the keyboard to detect presence of the hand;

following by the video sensors the contour of the detected present hand to detect positions of the fingers of the operator'"'"'s hand;

scanning by the video sensors along each of the fingers for determining locations of the tips of the fingers with respect to the keyboard to determine identification information of the keys touched with the fingers;

optically detecting with the video sensors each touch of the keys on the keyboard by the fingers of the operator'"'"'s hand wherein the step of optically detecting each touch comprises the step of determining speed of the finger in motion to touch the key of the keyboard as a three dimensional vector measurement along the X, Y and Z axes of motion of the finger;

entering the identification information of the keys touched with the fingers into the data processing system to represent the input data; and

monitoring subsequent positions of the fingers of the operator'"'"'s hand and comparing the input data corresponding to each subsequent monitored finger position to the identification information of the keys touched, and corresponding each monitored position of the fingers of the operator'"'"'s hand with individual key locations on the keyboard.

2. The method of claim 1, wherein the step of determining speed comprises the step of determining speed of the finger before a touch impact on the key.

3. The method of claim 1, wherein the step of determining speed further comprises the step of determining speed of the finger during a touch impact on the key.

4. The method of claim 1, wherein the step of determining speed further comprises the step of determining speed of the finger after a touch impact on the key.

5. The method of claim 1, further comprising the step of identifying each respective key on the keyboard as a location where the finger of the operator'"'"'s hand touching the respective key stops.

6. In a data processing system, a method of data input using video sensors for monitoring positions of an operator'"'"'s hand with respect to keys on a virtual keyboard optically produced on a surface and for subsequent and correlated data recognition by corresponding key locations on the keyboard to the monitored positions of the fingers of the operator'"'"'s hand, comprising the steps of:

touching the keys with fingers of the operator'"'"'s hand to input data;

scanning by the video sensors adjacent to the keyboard to detect presence of the hand;

following by the video sensors the contour of the detected present hand to detect positions of the fingers of the operator'"'"'s hand;

scanning by the video sensors along each of the fingers for determining locations of the tips of the fingers with respect to the keyboard to determine identification information of the keys touched with the fingers;

optically detecting with one of the video sensors each touch of the keys on the keyboard by the fingers of the operator'"'"'s hand, wherein the step of optically detecting each touch of each key includes the step of determining speed of each finger in motion for touching each respective key of the keyboard and detecting each touch providing the speed before each respective touch impact of each respective key exceeds a predetermined minimum value, the speed during the touch impact of each respective key exceeds the speed after the touch impact, and the speed after the touch impact of each respective key is less than the speed before a next key touch impact on the keys of the keyboard with the operator'"'"'s finger;

entering the identification information of the keys touched with the fingers into the data processing system to represent the input data; and

monitoring subsequent positions of the fingers of the operator'"'"'s hand and comparing the input data corresponding to each subsequent monitored finger position to the identification information of the keys touched, and corresponding each monitored position of the fingers of the operator'"'"'s hand with individual key locations on the keyboard.

7. A data processing system for keyboard data input and positional data correlation corresponding to keys on the keyboard, comprising:

a virtual keyboard generated optically on a surface for providing a presentation of a keyboard having a plurality of keys for touching with fingers from a hand of an operator;

video sensors for monitoring the keyboard and the fingers of the operator'"'"'s hand to generate image data representing positions in three dimensions of the fingers of the operator'"'"'s hand with respect to the keyboard and the keys;

means for processing responsive to the image data for generating input data representing identification information of the keys touched with the fingers of the operator'"'"'s hand; and

positional data correlation means for detecting an occurrence when the video sensors monitor that one of the fingers of the operator'"'"'s hand generates three dimensional image data corresponding to identification information of one of the respective touched keys of the keyboard.

8. A method of keyboard data input into a data processing system using a video sensor for determining vertical positions of an operator'"'"'s fingers with respect to keys on a keyboard surface, comprising the steps of:

hitting the keys with the operator'"'"'s fingers,

monitoring the keyboard and the operator'"'"'s fingers by the video sensor to detect a sequence of successive motions of each of the operator'"'"'s fingers in the vertical direction from the keyboard surface with respect to each of the operator'"'"'s fingers;

detecting the keys hit with each of the operator'"'"'s fingers based on the sequence of successive motions and the vertical positions of each respective operator finger; and

entering identification information of the detected keys of the keyboard into the data processing system to represent input data for correlation to subsequent hitting of the keys by the operator'"'"'s fingers.

9. In a data processing system, a method of data input using video sensors for monitoring positions of an operator'"'"'s hand with respect to keys on a virtual keyboard optically produced on a surface and for subsequent and correlated data recognition by corresponding key locations on the keyboard to the monitored positions of the fingers of the operator'"'"'s hand, comprising the steps of:

touching the keys with fingers of the operator'"'"'s hand to input data;

scanning by the video sensors adjacent to the keyboard to detect presence of the hand;

following by the video sensors the contour of the detected present hand to detect positions of the fingers of the operator'"'"'s hand;

scanning by the video sensors along each of the fingers for determining locations of the tips of the fingers with respect to the keyboard to determine identification information of the keys touched with the fingers;

optically detecting with one of the video sensors each touch of the keys on the keyboard by the fingers of the operators hand, wherein the step of optically detecting each touch comprises the step of determining speed of the finger in motion to touch the key of the keyboard during a touch impact on the key which includes the step of measuring the dynamic impact of the finger at the time of the touch of the finger of the operator'"'"'s hand to the keyboard surface;

entering the identification information of the keys touched with the fingers into the data processing system to represent the input data; and

monitoring subsequent positions of the fingers of the operator'"'"'s hand and comparing the input data corresponding to each subsequent monitored finger position to the identification information of the keys touched, and corresponding each monitored position of the fingers of the operator'"'"'s hand with individual key locations on the keyboard.

10. The method as defined in claim 9, where the method of measuring the dynamic impact of the finger, measures pressure of the tip of the finger on the touched key of the keyboard.