Intelligent robotic interface input device

• US 20100103106A1
• Filed: 07/08/2008
• Published: 04/29/2010
• Est. Priority Date: 07/11/2007
• Status: Active Grant

• Current Assignee: Hsien-Hsiang Chui
• Original Assignee: Hsien-Hsiang Chui
• Inventors: Chui, Hsien-Hsiang
• Granted Patent: US 8,552,983 B2
• Time in Patent Office: Days
• US Class Current: 345/166
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G09B 21/00 Teaching, or communicating with, the blind, deaf or mute audible presentation of material to be studied G09B5/04; devices or methods for replacing direct visual or auditory perception by another kind of perception A61F9/08, A61F11/04; audible indication of meter readings or of colour G01D7/12; watches for blind persons G04B25/02; speech analysis, speech recognition G10L; sound-recording or reproducing, per se G11B

Abstract

Universal Video Computer Vision Input Virtual Space Mouse-Keyboard Control Panel Robot has computer system use video vision camera sensors, logical vision sensor programming as trainable computer vision seeing objects movements X, Y, Z dimensions'"'"' definitions to recognize users commands by their Hands gestures and/or enhance symbols, colors objects combination actions to virtually input data, and commands to operate computer, and machines. The robot has automatically calibrated working space into Space Mouse Zone, Space Keyboard zone, and Hand-Sign Languages Zone between user and itself. The robot automatically translate the receiving coordination users'"'"' hand gesture actions combinations on the customizable puzzle-cell positions of working space and mapping to its software mapping lists for each of the puzzle-cell position definition and calibrate these user hand and/or body gestures'"'"' virtual space actions into entering data and commands to computer meaningful computer, machine, home appliances operations.

Claims

1. A vision space mouse-keyboard control panel Robot has computer system use video vision camera sensors, logical vision sensor programming as trainable computer vision seeing objects movements X, Y, Z dimensions'"'"' definitions to recognize users commands by their Hands gestures and/or enhance symbols, colors objects combination actions to virtually input data, and commands to operate computer, and machines.

2. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said has automatically calibrated working space into Space Mouse Zone, Space Keyboard zone, and Hand-Sign Languages Zone between user and itself.

3. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said automatically translate the receiving coordination users'"'"' hand gesture actions combinations on the customizable puzzle-cell positions of working space and mapping to its software mapping lists for each of the puzzle-cell position definition.

4. A vision space mouse-keyboard control panel Robot according to claim 3 wherein said the calibrate puzzle-cell position has define meaning that mapping on its software program for robot to transfer the virtual hands gesture actions into entering data and commands to operating computer, machines and the robot act as Universal virtual Space Mouse, virtual Space Keyboard, and virtual Remote Controllers.

5. A vision space mouse-keyboard control panel Robot according to claim 4 wherein said Calibrate these user hand and/or body gestures'"'"' virtual space actions into entering data and commands to computer meaningful operations moving cursor UP, Down, Left, Right, Clicks, typing Texts, Hand-Sign Languages reading, Symbol Character Writing, and Drawings and commands to operate machines, and home appliances.

6. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said computer vision consistently watching and to recognized user hands gesture movements coordinating with its define puzzle-cell positions of the virtual projecting working space zones that robot will automatically translate the receiving coordination users'"'"' hand gesture actions on the puzzle-cell positions'"'"' combinations and mapping to its software mapping lists for each the puzzle-cell position definition and calibrate these user hand and/or body gestures'"'"' space actions into computer meaningful operations such as Virtual Space Mouse input that moving cursor UP, Down, Left, Right, Left Clicks, Right Clicks, and also as Virtual Keyboard enter text, character input as to typing characters and function keys such as a, A, b, B, c, C, Backspace, Ctrl, Shift, Del, Enter key, computer operation commands.

7. A vision space mouse-keyboard control panel Robot according to claim 2 wherein said also able to provide Hand-Sign Languages reading from user'"'"'s hands and/or body gesture according to its preprogram listing of hand-sign languages gesture patterns and grammars, robot can recognize what words and/or commands that user wants to enter.

8. A vision space mouse-keyboard control panel Robot according to claim 7 wherein said can be enable symbolic characters writing such as Chinese characters, and drawing a picture into computer by user'"'"'s hand gestures'"'"' movements.

9. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said can be trained and taught to tracking on a specific object by recognize its shape, symbols and/or colors and optional embedded wireless sensors attached on the tracking objects that to enhance the object tracking reliable vision reading and also to fit user'"'"'s usage preferences, especially for those who have physical limitation special needs to operate the computer or machine.

10. A vision space mouse-keyboard control panel Robot according to claim 3 wherein said the puzzle-cell positions of Space Mouse, and Space Keyboard of the robot can be customized. The puzzle-cell position of space zone to allow user to reprogram the standard working zone positions of Space Mouse, and Space Keyboard to be customized the certain keys rearrangements, and setting up for the certain puzzle-cell position of working space zone for certain meaning of text and commands to represented.

11. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said can be integrated to Home Appliances Automation by having the robot install on the home, and the robot consistently watching for the owner to making commands by their hand gestures and/or voice commands (by speech reorganization software program) to operating each electric devices and turn ON/OFF individual lights at home.

12. A vision space mouse-keyboard control panel Robot according to claim 11 wherein said will simplify all the remote controllers for all of appliance at home into hand gestures commands.

13. A vision space mouse-keyboard control panel Robot according to claim 12 wherein said With a customized train robot to recognized a any particular wood stick that can become a universal remote controller of all appliances at the home as a Magic Stick Remote controller instantly.

14. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said Robot equipped microphone, sounds sensors, speech recognition software program to listening user'"'"'s voice commands, and equipped speaker to reading out text, article, to communicate with user. The optional reading out of the user'"'"'s each input character and commands as voice feedback that to aid users to know what key they entering;

in addition, the robot also equipped a Motor Vibrate Silent-Reading sub-robot module.

15. A vision space mouse-keyboard control panel Robot according to claim 14 wherein said equipped a Motor Vibrate Silent-Reading sub-robot module that comprise a micro controller as programmed brain, a 2 sections of vibrations surface for use to distinguished Long Short signal coding to get the reading Morse code text coding, 2 Seashell shape of Springs Coils attached on the each of motors to be spins (1 larger than the other) that will generate Long signal and short signal) and 2 motors (can be Step motors, or Servo motor, or DC motors), one motor for rotate short spin vibrations, and the other motors for rotate long spin vibrations to generate silent-reading Morse code and standard text coding for users especially for people who can not see and can not hear.

16. A vision space mouse-keyboard control panel Robot according to claim 15 wherein said is to benefit to everyone to use computers and machines without physical ability limitations and the proposal robot can improve current small keyboard space hard to typing problem on cell phone, portable devices;

in addition, the proposal robot can integrated into home automations and reduce multiple remote controllers, and provide handful assist for users who have physical ability limitations to be able to operate computers and machines easier like normal people do.

17. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said Initializes Sequence Steps to Training Vision-Tracking Objects Inputs. Logical vision tracking program can be trained to tracking on very small of finger'"'"'s movement gestures actions by lock on each individual specific object'"'"'s shape, size, its colors and/or embedded wireless sensors/LED Lights, laser beam lights on the objects that user fingers carry or wear or draw on.

19. A vision space mouse-keyboard control panel Robot according to claim 12 wherein said Virtual Working Space that robot automatically measurement and virtually project working space zones between user and itself.

22. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said The robot 1 equipped plurality video vision sensor cameras, sensors, and web cameras for X, Y, Z dimension tracking reading.

25. A vision space mouse-keyboard control panel Robot according to claim 2 wherein said Virtual Space Zone that user able to use hand gesture to moving the cursors position on the monitor, and perform Mouse click to drafting hold moving the page on the monitor screen Up, Down, Left and Right, accordingly to user hands gesture movements in X, Y dimension surface position, and user able virtually to Double Click mouse button, or Right click button by changing user hand gesture Z dimension surface position toward the robot as clicks actions.

27. A vision space mouse-keyboard control panel Robot according to claim 2 wherein said The Virtual Space Keyboard Zone has calibrated into puzzle-cell positions. As the standard keyboard keys arrangements, the first right side puzzle-cell position key of G-Point is representing “

H”

key, and the first left side of G-Point is representing “

G”

key.

34. A vision space mouse-keyboard control panel Robot according to claim 7 wherein said The robot'"'"'s logical vision tracking program tracking confirm each finger'"'"'s XYZ position values relative distant with robot'"'"'s Vision-G-Point center to understand where each finger hold in, which finger to point out as those standard Hand-Sign Language gestures.

37. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said Initializes Sequence Steps to Anti-Unintentional-Unauthorized-Commands Inputs.

38. A vision space mouse-keyboard control panel Robot according to claim 37 wherein said the user'"'"'s initialized sequence gesture actions will wake up the sleeping mode robot and start to accept user command.

39. A vision space mouse-keyboard control panel Robot according to claim 38 wherein said from user initialized sequence gesture actions, robot able to determine that it is not just someone or something walking through its virtual working space zones. The robot can determine the user has the intelligent/knowledge to use vision commands and intent to enter data and commands.

40. A vision space mouse-keyboard control panel Robot according to claim 39 wherein said initialized hand/body gestures sequences actions, the actions will help robot to measurement of user'"'"'s proper working space, and setting the adjusting working space size and the 360 degree edge of working space (X, Y) and proper z dimension working space range, according what user'"'"'s initial hand around space length range.

41. A vision space mouse-keyboard control panel Robot according to claim 40 wherein said From Authority/robot owner User will perform presetting personal unique gesture action sequences, Position Translate Program recognize the sequence actions that match with its presetting authority gestures sequences, knowing that the user is the owner, the master has the permission to use, setting any robot'"'"'s administrate permission commands to program the robot'"'"'s behavior, functions, and has right to train the robot to recognized focus on special object. The initial sequence steps can act as robot operation lock gates that user has the master permission, and/or has use the robot, or no permission to use it at all.

44. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said The robot has a Morse code or standard text coding program it can convert the text of article into Morse code'"'"'s long and short signals and robot will sending commands to smart phone to operating micro controller to turn on long vibrate motor or short vibrate motor in Morse code series long short vibrate signals. In series actions of motor long and short vibrate Morse code signals as result the sub-robot module to generate a represent signal of a text or words sentences for user who carry it to silent reading the article info from robot.

45. A vision space mouse-keyboard control panel Robot according to claim 44 wherein said vibrate signal generation option methods, each motor can attached a stick in horizontal position and when motor turn the stick slightly touch on users and back to the horizontal position, one for short signal and one for long signal.

46. A vision space mouse-keyboard control panel Robot according to claim 44 wherein said vibrate signal generation option methods the sub-robot module can simple to divide into 2 sections on the left half side vibrate surface is represent short signal, and on the right half side vibrate surface is represent long signal.

47. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said Home-base type of Virtual Space Mouse-Keyboard Control Panel Robot that equipped video web camera for X, Y dimension tracking reading, video web camera for Z dimension tracking reading. The robot equipped plural type of sensors;

the connection method means between connection of sensors, and videos to Robot can be wire or wireless sensors network and video sensor cameras network.

48. A vision space mouse-keyboard control panel Robot according to claim 47 wherein said Puzzle-Cell Position Coordination Translate Program will automatically translate the receiving XYZ coordination of users'"'"' hand gesture actions puzzle-cell position of working space. Robot'"'"'s Virtual Space Cell Position of Text &

Command Conversion Respond Action Map Programming List and mapping match this translated virtual space cell position that from user with robot'"'"'s software Virtual Space Cell Position Text &

Command Respond Map Programming List for each of the puzzle-cell position definition respond actions

49. A vision space mouse-keyboard control panel Robot according to claim 48 wherein said Robot has the ability to be able to calibrate those user'"'"'s hand gestures virtual space actions convert into entering the selected text and command to execute computer meaningful operations to appliances such as TV, computer, Laptop, Light, iRobot vacuum Roomba, electronic door lock.

50. A vision space mouse-keyboard control panel Robot according to claim 49 wherein said Robot can integrate with the Home Appliance Automation System and user, home owner can use their gestures action to control operating all of appliances at home.

51. A vision space mouse-keyboard control panel Robot according to claim 50 wherein said the plural video cameras and plural types of sensors that installed on each rooms of the property as video camera networking and sensors networking. The video camera can install with a motor that has holder to hold the video camera that controlled by robot, the robot'"'"'s vision sensors will tracking follow the user special object where user moving to activate motor to tracking rotate the video camera changing direction to aiming video camera user and specific tracking object where they are and waiting for any command from the user.

52. A vision space mouse-keyboard control panel Robot according to claim 51 wherein said the user command robot to read the article in silent-reading functions, the robot will remote control to Motor Vibrate Silent-Reading sub-robot module and transmit the Morse coding for each words and sentence through wireless with Bluetooth protocol to connect smart phone that adapted on the sub-robot and to operate the micron controller to spin the Long, Short motors to generate Long Short vibrate Morse Code for the user to read the article from PC-base robot.

53. A vision space mouse-keyboard control panel Robot according to claim 51 wherein said the user need to robot speak machine voice feedback, then the robot activate the speaker and with speech recognize software program to make text-speech conversation and read out the machine voice to communicate with user. The robot able to hear user voice commands as well from microphone, sound sensors.

54. A vision space mouse-keyboard control panel Robot according to claim 52 wherein said total tracking vision especially for Home-base type robot, video camera install on each room with motor that hold it, and the motor spins directions to be control by the robot vision sensors and will able to rotate video camera to following where the user, specific tracking object on the room.

55. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said the robot able to use variety types of sensors that installed to be a sensor networks and different types of sensors reading signals can be unify into XML, a same signals format, which can be used by the robot for advance intelligent robot performance.

56. A vision space mouse-keyboard control panel Robot according to claim 1 wherein said the features of the Universal Video Computer Vision Input Virtual Space Mouse-Keyboard Control Panel Robot built into a single microchip.

57. A vision space mouse-keyboard control panel Robot according to claim 56 wherein said the virtual space mouse-keyboard control panel robot'"'"'s translate space gestures actions with software mapping key listing method into entering data and commands to operating computer, This method can be built/embedded into a microchip, or a processor, or a video processor that contains these four sections of intelligent virtual space-commands software programs/scripts that demonstrated above, and with three working space calibrate alignment arrangement standard for create virtual Space Mouse, Virtual Space keyboard, and Hand-Sign Languages patterns.

58. A vision space mouse-keyboard control panel Robot according to claim 57 wherein The 4 sections of intelligent virtual space-commands conversation software are 1. The Logical Vision Tracking Program, 2. X, Y, and Z surface direction following program, 3. The X, Y, Z dimension value program, and 4. X, Y, and Z Position Translate Program.

59. A vision space mouse-keyboard control panel Robot according to claim 57 wherein said the processor contains three working space calibrate alignment arrangement standard for Mouse, keyboard, and Hand-Sign Languages patterns to automatically initialize the virtual working space of Virtual Space Mouse, Virtual Space Keyboard, and Hand-Sign Languages Zones

60. A vision space mouse-keyboard control panel Robot according to claim 57 wherein said the Virtual Space Mouse-Keyboard Control Panel Robot microchip can install on to any computer, machine, and appliance at home, and able to connect the video vision camera sensor, running by Windows XP, CE embedded, or Linux operation software to provide the virtual space mouse, space keyboard on those computer, and machines.

18. A vision space mouse-keyboard control panel Robot according to claim 18 wherein said has trainable computer vision to recognize the user'"'"'s hand gestures commands with specific symbol shape, size, color, and/or optional embedded wireless sensors, LED lights, Laser beam Lights for reliable vision tracking reading to remote control all of the appliances at user'"'"'s property.

42. A vision space mouse-keyboard control panel Robot according to claim 18 wherein said Initializes Sequence Steps to Training Vision-Tracking Objects Inputs, user using 2 steps training object tracking, the first step, the user 11 has wear special objects start moving object around working space, and the second step, the user move the objects from the high traffic area of user working space zones. During the two steps of special object training vision tracking, the robot logical vision tracking program will compare the video frames in series, the program will match frames and know to filter out the background of image signal that don'"'"'t move, logical vision tracking program compare video frame know what objects signals about its particular shape, size, color, and/or embedded wireless sensors/LED Lights, laser beam lights on the object indications that chance XYZ dimension value every time. The logical vision-tracking program has learned the special object signals to be tracking.

43. A vision space mouse-keyboard control panel Robot according to claim 42 wherein said the logical vision-tracking program can also be hard coding manually programming by user as well for tracking on special object. The user can directly program at robot'"'"'s and set coding tracking on what shape of object, what size, what colors and/or if there is embedded wireless sensors, LED Lights, laser beam lights on the objects any indications to be vision tracking on. The logical vision-tracking program will base on the input object definition and looking only the object that matched

20. A vision space mouse-keyboard control panel Robot according to claim 20 wherein said When the robot sensor detect user, and robot use video web cameras, video vision camera sensor to measure of user height and width, and automatically calibrated virtual working space, robot adjusting the distant between user and itself to projecting the Virtual Space Mouse zone, Virtual Space Keyboard, and Hand-Sign Languages Zone.

21. A vision space mouse-keyboard control panel Robot according to claim 21 wherein said the working space can be selected and choose to work on either one of these 3 space function zones, or to have divide whole working space into 3 zones for space mouse, keyboard, and Hand-Sign Languages zone together.

23. A vision space mouse-keyboard control panel Robot according to claim 23 wherein said logical vision tracking program tracking will changing X surface direction, Y surface direction, Z surface direction to virtual space XYZ value relative distant with robot'"'"'s Vision-G-Point center.

24. A vision space mouse-keyboard control panel Robot according to claim 24 wherein said Position Translate Program convert the new tracking position space XYZ value into its mapping computer operation actions to auto execute the commands that by user hand gestures actions.

26. A vision space mouse-keyboard control panel Robot according to claim 26 wherein said The user can mimic Regular physical mouse operating actions in one hand in Virtual Space Mouse zone, the robot able to precisely tracking fingers X, Y, Z gesture movements and perform the Virtual Space Mouse functions.

28. A vision space mouse-keyboard control panel Robot according to claim 28 wherein said The robot sensor detect user, and robot use video web camera to measure of user height and width, and automatically calibrated working space, robot will virtual project the dimensions'"'"' axis G-Point that represent the center point of whole working space in relative 3D level user working space of X dimension surface, Y dimension surface, and Z dimension surface. The user'"'"'s hand gesture'"'"'s X, Y, Z space positions will be base on the relation distant of the G-Point.

30. A vision space mouse-keyboard control panel Robot according to claim 28 wherein said When user select “

H”

key”

, The robot'"'"'s logical vision tracking program tracking will accept user hand gesture position change Z surface direction toward the robot, the Z dimension value will be add −

1 that confirm the user selected key position changing X surface direction, and the X dimension value will be add 1 since the “

H”

key is relative one right-side key distant with robot'"'"'s Vision-G-Point center.

29. A vision space mouse-keyboard control panel Robot according to claim 29 wherein said projecting the mimic physical keyboard alignments angles relation and arrange the puzzle-cell positions in keyboard style as Virtual Space Keyboard.

31. A vision space mouse-keyboard control panel Robot according to claim 31 wherein said The robot'"'"'s logical vision tracking program received dimension X, Y changing value will be automatically to be translated by its Position Translate Program into keyboard mapping listing, the new X tracking value will be match on “

H”

key and display the “

H”

character on the monitor.

32. A vision space mouse-keyboard control panel Robot according to claim 32 wherein said The two steps Z value selections method, Example, Use Space “

Shift”

key or any special function keys, two steps to accept the Z surface direction, the user punch out on left hand on “

Shift”

key position, the Z dimension value will be add −

1, and robot'"'"'s Position Translate Program map value into keyboard mapping listing aware of that is a meaningful puzzle space as “

Shift”

key position, and wait for the Second selection, then user move right hand to the “

A”

key position and then use left hand punch out toward robot further again to make confirm key selection and the robot logical vision tracking program accept the Z surface direction, the Z dimension value will be add −

1 to be −

2, and its Position Translate Program into keyboard mapping listing aware of that is double “

Shift”

key twice will confirm the select key, and the new X surface direction, and the X dimension value will be add −

5 relative distant with robot'"'"'s Vision-G-Point center and the new Y surface direction, and the Y dimension value will be add 0 relative distant with robot'"'"'s Vision-G-Point center, and its Position Translate Program into keyboard mapping listing aware of that is a meaningful puzzle space as Capital “

A”

key.

33. A vision space mouse-keyboard control panel Robot according to claim 33 wherein said The same 2 steps special function keys selection principal method can apply to using “

Ctrl”

, “

Alt”

, Special function keys, “

!”

, “

@”

, “

#”

, “

$”

, “

%”

, “

̂

”

, “

&

”

, “

*”

, “

(”

, “

)”

, “

{”

, “

}”

, “

_”

, “

+”

all of keys that require two steps selection method.

35. A vision space mouse-keyboard control panel Robot according to claim 35 wherein said The Hand-Sign 360 degree XYZ Position Translate Program will match the series tracking value to get the specific Hand-Sign words that user'"'"'s Hand Sign Language gesture.

36. A vision space mouse-keyboard control panel Robot according to claim 36 wherein said train robot'"'"'s logical vision tracking program to recognize a special object such as a sharp point of a pen. The user hold the sharp point of the pen face to robot, and start to move around the pen as it writing word on air or drawing a picture on air, the robot watch each video frame and mark the sharp point of the pen XYZ value, and then update the value to monitor or a painting software, the series frames signals xyz values will compose into meaning symbolic character writing or a unique drawing picture from user.