Intelligent robotic interface input device

• US 8,552,983 B2
• Filed: 07/08/2008
• Issued: 10/08/2013
• Est. Priority Date: 07/11/2007
• Status: Expired due to Fees

• Current Assignee: Hsien-Hsiang Chiu
• Original Assignee: Hsien-Hsiang Chiu
• Inventors: Chiu, Hsien-Hsiang
• Primary Examiner(s): Wong, K.
• Application Number:
  US12/311,239
• Publication Number:
  US 20100103106A1
• Time in Patent Office: 1,918 Days
• Field of Search: None
• US Class Current: 345/166
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

An intelligent object tracking and gestures sensing input device is operative to translate hand gestures of a user into data and commands for operating a computer or various machines. It is provided with video web cameras video vision camera, sensors. A logical vision sensor program in the device measures movements of the user'"'"'s hand gesture into X, Y and Z dimension definitions. It defines a working space spaced therefrom into virtual space mouse zone, space keyboard zone, and hang sign language zone. It automatically translates the change of coordinates of the user'"'"'s hand in puzzle cell positions in the virtual working space into the data and commands. Objects having enhance symbols, colors, shape, and illuminated lights are attachable on the user'"'"'s hand to provide precision input.

Claims

1. A computer program product comprising a non-transitory computer usable medium having control logic stored therein for causing an input device to track and transform hand gestures into input commands for controlling a computer or electrically operated machine, said control logic comprising:

First computer readable program code for processing X, Y and Z coordinates dimension data by recording a width and a height of a user to determine the X and Y coordinates and a distance of the user to determine the Z coordinates

Second computer readable program code for providing input data and commands for executing various operations of computer or electrically operated machine;

Third computer readable program code for establishing a virtual working area by the user'"'"'s width and height, wherein a center point of the virtual working area relative to the user is established;

Fourth computer readable program code for establishing a virtual space keyboard zone, wherein keys in the virtual space keyboard zone are positioned relative to the center point of the virtual working area; and

Fifth computer readable program code for establishing a virtual hand-signal language zone, wherein X, Y, Z finger coordinates are established relative to the center point of the virtual working area to establish hand gestures.

2. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for creating puzzle cell positions in each of said virtual space mouse zone, virtual space keyboard zone, and virtual hand sign language zone; and

ninth computer readable program code for correlating said puzzle cell positions with a mapping list to determine a command to be executed in said computer or electrically operated machine by said hand gestures of said user.

3. The non-transitory computer readable program product of claim 2 wherein said mapping list is a listing of commands for executing various operations of said computer or electrically operated machine.

4. The non-transitory computer readable program product of claim 2 wherein said commands are operation commands of an operating computer connected to said input device and said logic vision tracking program provides to said operating computer cursor operations including Up, Down, Left, Right, mouse clicks, typing of a selected text, reading of hand sign language, and writing of a symbol character, and drawing of a picture in said operating computer.

5. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for tracking said user located in said space keyboard zone;

ninth computer readable program code for defining a plurality of puzzle cell positions in said space keyboard zone; and

tenth computer readable program code for receiving changes in coordinates data made by said hand gestures of said user to correlate with said mapping list of commands for actuating a selected computer operation.

6. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for translating hand gestures made by said user located in said hand-sign language zone; and

ninth computer readable program code for correlating said gestures with said listing of hand sign language gesture patterns to provide a selected command for operating said computer or electrically operated machine.

7. The non-transitory computer readable program product of claim 1 further comprising eighth computer readable program code for translating hand actions of said user for writing Chinese characters and drawing pictures in a computer.

8. The non-transitory computer readable program product of claim 1 further comprising eighth computer readable program code for tracking shape, symbols and colors of a specific object located at said virtual working space.

9. The non-transitory computer readable program product of claim 2 wherein said puzzle cell positions are customizable to represent a selected key arrangement operative for executing command operation of said computer or electrically operated machine.

10. The non-transitory computer readable program product of claim 1 further comprising eighth computer readable program code for receiving voice commands from said user.

11. The non-transitory computer readable program product of claim 1 further comprising eighth computer readable program code for initializing sequence steps of tracking small movements of fingers in said user'"'"'s hand gestures.

12. The non-transitory computer readable program product of claim 11 further comprising ninth computer readable program code for providing dimensional measurements of said virtual working space.

13. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for measuring said user'"'"'s height and width; and

ninth computer readable program code for automatically calibrating the distance from said user to said virtual working space to establish said virtual space mouse zone, virtual space keyboard zone, and virtual hand sign language zone.

14. The non-transitory computer readable program product of claim 13 further comprising tenth computer readable program code for enabling said input device to operate with all three zones simultaneously.

15. The non-transitory computer readable program product of claim 14 further comprising eleventh computer readable program code for tracking changing X surface direction, Y surface direction, and Z surface direction of said user'"'"'s hand gestures in establishing a virtual space XYZ coordinate value relative to the distance of movement of said hand gestures from a vision G-point center designated in said virtual working space.

16. The non-transitory computer readable program product of claim 15 further comprising twelfth computer readable program code for converting said XYZ coordinate value into a selected command for operating said computer or electrically operated machine by said hand gestures of the user.

17. The non-transitory computer readable program product of claim 15 further comprising:

twelfth computer readable program code for determining physical keyboard alignment angles relative to said G-point; and

thirteenth computer readable program code for establishing puzzle cell positions in said virtual space keyboard zone to form a virtual space keyboard.

18. The non-transitory computer readable program product of claim 15 further comprising:

twelfth computer readable program code for confirming change in Z surface direction by adding a −

1 value to the Z dimension value when said user makes a gesture by placing a hand at a puzzle cell position to make a key selection of an H key in said virtual space keyboard; and

thirteenth computer readable program code for adding a value of 1 to the X dimension value to designate distances of said H key from said G-point.

19. The non-transitory computer readable program product of claim 18 further comprising:

fourteenth computer readable program code for translating the change in values of said X dimension and Z dimension to compare with key locations in said keyboard mapping list; and

fifteenth computer readable program code for executing said key selection by displaying an H character on a monitor of said computer.

20. The non-transitory computer readable program product of claim 19 further comprising sixteenth computer readable program code for tracking and confirming XYZ position coordinate values relative to the distance from said G-point of a finger in each hand of said user for determining desired operations made by standard hand sign language gestures.

21. The non-transitory computer readable program product of claim 20 further comprising seventeenth computer readable program code for matching series of tracking values representing specific hand sign words for establishing word selections made with hand sign language gestures by said user.

22. The non-transitory computer readable program product of claim 21 further comprising:

eighteenth computer readable program code for recognizing movement of a sharp point of a pen wherein said user holding said pen and pointing said sharp point towards said robotic input device is able to make movement of said sharp point for writing a word, character, and picture drawing selectively; and

nineteenth computer readable program code for translating the XYZ position value of the movement of said sharp point in providing input data for drawing said word, character, and picture respectively in said computer.

23. The non-transitory computer readable program product of claim 1 further comprising eighth computer readable program code for comparing sequential steps of hand gestures of said user with a memory listing of acceptable sequential steps of hand gestures to determine whether said hand gestures are performed by an acceptable user.

24. The non-transitory computer readable program product of claim 23 further comprising ninth computer readable program code for waking said input device from a sleeping mode prior to accepting further hand gestures from said user to execute operations of said computer or electrically operated machine.

25. The non-transitory computer readable program product of claim 24 further comprising tenth computer readable program code for processing and executing tracking inputs in the training of said input device when tracking a special object, said inputs having two steps wherein in a first step said user moves a hand in said working space with said special object having a particular shape, size, color, embedded wireless sensors, LED lights, and laser beam lights selectively adapted on said hand, and in a second step said user moves said special object in a high traffic area of said working space zone, said logical vision tracking program being operative to compare series of video frames obtained in said two steps to determine said tracking operation of said special object.

26. The non-transitory computer readable program product of claim 25 further comprising eleventh computer readable program code for enabling the hard code manual programming by said user for tracking said special object, in said hard coding manual programming said user setting tracking codes of said special object having said shape, size, color, embedded wireless sensors, LED lights, and laser beam light selectively, said logical vision tracking program being operative by said tracking codes for tracking an object matching said tracking codes.

27. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for converting text from a computer into long and short signals according to standard Morse code; and

ninth computer readable program code for enabling the transmission of said signals to a smart phone to operate a micro controller for operating a long vibrate motor and a short vibrate motor, a reading module being coupled to said micro controller and receiving long and short signals from said long vibrate motor and said short vibrate motor respectively to generate long and short Morse code signals representing article text for the silent reading of said article text by said user.

28. The non-transitory computer readable program product of claim 1 further comprising:

eighth computer readable program code for receiving voice commands from said user;

ninth computer readable program code for reading out text;

tenth computer readable program code for carrying out voice communication with said user; and

eleventh computer readable program code for providing silent reading of text and article in Morse code represented by hand gestures by said user.

29. A system for tracking and transforming a user'"'"'s hand gestures into input commands for controlling a computer or electrically operated machine, comprising,

An input device;

At least one video vision camera, in operable communication with said input device;

At least one vision sensor, in operable communication with said input device;

At least one web camera, in operable communication with said input device, wherein said web at least one web camera, along with said at least one vision sensor, are configured to automatically scan a selected work space for tracking an object;

An automatic motor in operable communication with said video vision camera wherein said motor is controlled by said input device to enable said video vision camera to track and follow said user continuously;

A central control computer, in operable communication with said input device, said video vision cameras and vision sensors;

Computer readable program code for processing X, Y and Z coordinates dimension readings by recording a width and a height of a user to determine the X and Y coordinates and a distance of the user to determine the Z coordinates, wherein a center point is determined based on the X, Y, and Z coordinates and the hand gestures is based on distances between the X, Y, and Z coordinates and the center point;

A microphone, in operable communication with said input device;

At least one sound sensor in operable communication with said input device.

30. The system according to claim 29 further comprising wireless sensors attached to said object to enhance tracking of said object to facilitate operation of said computer or electrically operated machine by a person having physical disabilities.

31. The system according to claim 29 further comprising a wood stick mounted on said input device wherein said wood stick is configured to execute command operations in response to hand gestures from said user.

32. The system according to claim 29 further comprising a reading module comprising:

two sections of vibration surfaces, wherein said surfaces are configured to distinguish between long and short signal codes for reading a text in Morse code; and

two seashell shape spring coils attached on two spin motors, wherein said coils are configured to generate said long and short signal codes to facilitate silent reading of Morse code and standard text coding for a blind and deaf person.

33. The system according to claim 29 wherein said input device is configured to be incorporated into an electrical unit to facilitate operation of said electrical unit by using said user'"'"'s hand gestures.

34. The system according to claim 29 further comprising:

A long vibrate motor, in operable communication with said reading module, wherein said long vibrate motor is configured to receive long signals to generate long Morse code signals representing article text for silent reading of said text by said user; and

A short vibrate motor, in operable communication with said reading module, wherein said short vibrate module is configured to receive short signals to generate short Morse code signals representing article text for silent reading of said text by said user.

35. The system according to claim 34 further comprising a stick attached to said long vibrate motor and said short vibrate motor respectively whereby said stick moves back and forth from a horizontal position to contact said user wherein each said contact represents a Morse code signal.

36. The system according to claim 29 wherein said reading module is divided into two sections with a left half side vibrate surface representing a short signal, and a right half side vibrate surface representing a long signal.

37. The system according to claim 29 further comprising:

a video web camera, in operable communication with said input device and configured to track X and Y dimension readings;

a second video web camera, in operable communication with said input device and configured to track Z dimension readings; and

a plurality of various types of sensors within said input device, said sensors being in operable communication with said first video web camera and said second video web camera.

38. The system according to claim 29 further comprising speaker in operable communication with said input device, wherein said speaker is configured to operative provide voice feedback to said user, provide text speech conversation and read out text for said user.

39. The system according to claim 29, wherein said microphone and sound sensor are both configured to receive voice commands from said user.