Gesture Interface Robot

• US 20160350589A1
• Filed: 05/27/2015
• Published: 12/01/2016
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Hsien-Hsiang Chiu
• Original Assignee: Hsien-Hsiang Chiu
• Inventors: Chiu, Hsien-Hsiang
• Granted Patent: US 9,696,813 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

Gesture Interface Robot auto measure work zone to establish virtual puzzle cell locate in user gesture comfortable area where easy moving hands gesture to push out select virtual keys, prevent injury. Robot drawing virtual keyboard images graphic that display each puzzle cell command, and video vision sensing detect user'"'"'s hand location that on its virtual keyboard in the workspace zone. Special unique gesture hand sings design for enhances hand control virtual key selections. Push hand z dimension distant divide into 3 select zones. Real-time display highlight selected key graphic image to visual indicate for user to see. Using UIRT cable sending IR signal remote control computer, machine. Instantly generate virtual puzzle cell keyboard or controller that user selected. Introduce new gesture interface method, a Touch Screen Mouse combine with Puzzle Cell virtual keys in sandwich layers gesture zones method for operate computer keys and mouse operations, robot.

Claims

1. The completed working example model of Gesture Interface Robot (GIR) intelligent create comfortable hand gesture moving and push area for user, prevent injury, sandwich virtual control layers zones and able to sending Infrared signal to control other computer, machine, intelligent robot by simple hand gesture click select virtual puzzle cell command keys use components includes,Main Computer 1Video vision sensor module can be use Microsoft Kinect sensor. Build in Universal Infrared Receiver Transmitter (UIRT) 14 to this video sensor module as addition IR remote control features to physical operate machines.Micro Controller Board 21 can use Arduino board.Varity of motors modulesUSB-Universal Infrared Receiver Transmitter (UIRT)Display monitor

2. Gesture Interface Robot according to claim 1 Optional select to equipped wireless network equipments such as Bluetooth network card, Wi-Fi network card, all wireless network protocol card devices, TCP/IP, Internet Protocol such as Xbee, Ethernet, Wify, Bluetooth, Cell Phone channel 3G, 4G, GSM, CDMA, TDMA . . . etc, space telecommunication channel, satellite channels. Gesture Interface Robot according to claim 1 Option select to equipped Display monitor devices such as display monitor, image Projector and wireless network (example TCP/IP, or Bluetooth) display monitor glass. Gesture Interface Robot according to claim 1 Main Computer power source, wall power plug, and portable power source can be rechargeable batteries, solar cell, fuel cell, rotation generator, wind turbine, thermo electron generator (TEG), . . . etc. to regenerate electric power for robot to moving and operation. Gesture Interface Robot according to claim 1 Optional select to equipped mobile motor wheel platform, equipped motors wheels for control motor rotation direction and speed. Robots all components can be place on platform and robot is able to using video vision function to drive itself, moving around.

3. Gesture Interface Robot according to claim 1 Main Computer 1 use as Robot'"'"'s brain to process video image, The user body part joins location 3 dimension X, Y, Z values can be program using Microsoft visual C# program 4, (or VB), to calling Kinect and other system assemble libraries, and enable Kinect sensor to reading user joint values in the program. Gesture Interface Robot according to claim 1 These basic video sensor reading user'"'"'s 3D body joint values are available now, therefore we can write a specific puzzle cell map virtual keyboard control program (PCMVKCP) 3 that transform the basic 3D joint value, intelligently measuring calibrate into a new gesture interface input work space zone and establish puzzle cell virtual keyboard into the zone. So, user is able to moving hands, and point out to click virtual keys. Those enable Kinect sensor functions to reading joints values can coding into The (PCMVKCP) program 3, and the program 3 can be a class program (example:

MainWindow.xaml.cs) that include in the Microsoft Visual Studio C#4 as one project and build into 1 project solution, prefer in WPF Application type project, so all the video Kinect sensor reading value is available for (PCMVKCP) program to use them in real time programming, create dynamic user graphic interface.

4. Gesture Interface Robot according to claim 1 Gesture Interface robot use vision 3 dimension, X, Y, Z body parts value for robot vision puzzle cell map virtual keyboard control program (PCMVKCP) 3 to be able create work zone, establish puzzle cell map virtual keyboards in comfortable hand moving and push gesture area, prevent injury, provide real-time user hands location, convert to puzzle cell position, then match puzzle cell row-column to match with its puzzle cell command map list, transfer the cell position to computer command, and sending command to automation program 2 (such as EventGhost) to run pre record macro script to execute command such as display type a text, running a computer program, sending Infrared signal to remote control TV, DVD, or another computer to typing, mouse movement, mouse clicks, running computer program, internet browser . . . etc computer operations.

5. Gesture Interface Robot according to claim 1 In main computer 1 includes web server function 41, such as IIS server and can establish inter server-client network, DNS server, TCP/IPURL, namespace . . . etc. web site hosting, provide HTML, XMAL, scripting functions. When (PCMVKCP) program 3 can activate a web Browser, sending a web page URL include a specific text code, when particular web page is being running and open, the automation program 2 (such as EventGhost) detect the particular text code trigger, it will trigger the macro action in the folder. Example “

http;

//localhost;

8000/index.html?HTTP.KEYS_MU”

in Browser URL and enter. The web page activate link to trigger automation program EventGhost trigger event (KEYS Folder, MU event), and exercise the MU Marco script to sending out Infrared Signal to control another computer to move its mouse position UP small distant, if “

MU8”

, then move its mouse position UP large distant, if “

ML225”

, then move its mouse position 225 degree small distant. if “

ML225-8”

, then move its mouse position 225 degree, 8 times of small distant.

6. Gesture Interface Robot according to claim 4 comfortable hands moving and push out area, Gesture Interface Robot include vision (PCMVKCP) program can use these video sensor-reading values to create a perfect Workspace zone according to the user body measurements. The (PCMVKCP) program will assign a virtual center point on user, which is prefer to be should center joint point. The prefer Workspace zone width length is total length of each shoulder length×

1.5 (1.5+1.5=3), and the prefer Workspace zone height length is total length of Shoulder Center to Head top×

2. The Workspace zone will be tracking in front of user accordingly User'"'"'s should center joint point 79;

therefore when user walking or moving, the Workspace zone is always at the same place in front of user. If user walking the video view able area, the software keeps digital tracking, and when user walking out the video view area edge, then (PCMVKCP) program, will activate intelligent motor module rotate video sensor to follow aiming on user.

7. Gesture Interface Robot according to claim 1 create the Workspace zone size is defined, then (PCMVKCP) program will divide the Workspace zone into Puzzle Cell Row-Column formation. Gesture Interface Robot t Robot according to claim 11 he selection click zone z dimension user hands gesture click detect space is divided into 3 selection mode zones. 1st selection unlock selected key gate zone, 2nd moving to select virtual key zone, and 3rd select click zone.

8. Gesture Interface Robot according to claim 1 Gesture Interface Robot is using puzzle cell mapping method, Gesture Interface Robot is acting like a graphic image (Picasso) painter, robot program draw the graphic picture of virtual control panels keys. On the display monitor, the puzzle cell virtual control panel keys can be drawing a grid image of rows and columns cells and tilt with TextBlock field, then fill in text word to TextBlock field on each grid row and column cells on the graphic image as command, therefore inside the program function those text command words can be coding arranged into a 2 dimension array text strings, then loading each text word into row and column cells, so that display on the graphic puzzle cell image and virtually have assign on user'"'"'s working space zone. Gesture Interface Robot according to claim 13 Gesture Interface Robot can create any keyboard and control panel that user wanted includes computer QWERT keyboard, ABC, 123, Function keys, Symbols keys, Mouse keys, program keys, and create virtual control panel for machines such as TV, DVD, Sirius radio, Light, SLOT machine, Space Telescope, Disco DJ machine, Disco Light, virtual 3D world avatar, Mar'"'"'s Rover, Intelligent Robot. Gesture Interface Robot according to claim 14 Robot vision program draw virtual puzzle cell map keys control panel graphic image on display monitor and highlight select keys as visual indication, so use know which keys they are selected.

13. Gesture Interface Robot according to claim 1 has special interface arrangement, the virtual puzzle cell map keyboard, control panel, prefer special interface section arrangement, can be divide into Left and Right hand zones, The center area of virtual puzzle cell map keys control panel graphic image is reserved to place a real time video image that show user actions. So user can see themselves and all the control virtual keys together on monitor, this special virtual gesture interface arrangement make good visual feedback indication controls and easy for eyesight during for user operation.

14. Gesture Interface Robot according to claim 1 able to listening voice command, and speak voice feedback. If user say keyword the “

LOOK UP”

, “

LOOK RIGHT”

, “

LOOK LEFT”

, “

LOOK DOWN”

keys, robot will control its motor module to rotate its video sensor to turn UP, RIGHT, LEFT or Down direction, if user call Robot'"'"'s name, robot will determine sound source and automatic locate the user location and rotate motor to project video sensor view on the user.

15. Gesture Interface Robot according to claim 13 Puzzle Cell Morse code Keyboard is useful for poor eyesight user, and blind user to enter command by simplest gesture action “

Di”

, “

DHA”

actions to control machines.

18. Gesture Interface Robot according to claim 1 optional equipped the enhanced wireless select key indication device that wear on user hand palm, arms, or user body. The wireless indication device has 2 styles, 1^st style includes micro controller, Bluetooth, LED light, vibration motor and power source with flexible belt that can tight hold on hand palm. The 2^nd style include micro controller, wireless Wi-Fi, TCP/IP network card, LCD display screen, vibration motor, power source, watch belt to hold the device on hand. When user push hand out, program will sending wireless network signals to device to signal display which selection keys;

for example by blinking LED light in MorseCode signals, and/or using vibration motor to make long-short vibrations MorseCode signal, so user don'"'"'t need to watch display monitor to know what keys they select, this feature especially useful for poor eyesight, and blind users. The LCD screen can display real time monitor content, see the puzzle cell map image.

19. Gesture Interface Robot according claim 1 equipped a mobile platform, example using micro controller board to control varieties motors, robot main computer vision program can intelligently control these motors rotation;

as result, robot intelligently driving itself moving around and able to control moving its display projector direction to project puzzle cell keyboard images on any surface. Gesture Interface robot equipped micro controller board such as Arduino can programming the Micro controller and connect the Arduino board by COM port that connect to the (PCMVKCP) program in main computer, so the robot vision program can intelligent sending value string to Adruino board to rotate its motor direction, control speed, intelligently. The motor module can be use for video sensor Tilt, and Pan rotation, robot body movement, neck, arms, legs, and mobile wheels. Gesture Interface robot according to claim 28, the varieties of motors control modules can use to build into robot'"'"'s neck, body, arms, hands, legs, so robot can be build as human shape, physical body movement ability with Gesture Interface Robot puzzle cell map function. The Gesture Interface Robot becomes the communication bridge between human and intelligent robot machine world.

20. Gesture Interface Robot according to claim 1 Wireless Display Glass that has network protocol equipment includes wireless network card equipment, video image process card equipment, connect with projector, power source, and wireless server-client program to connect with robot, and robot send the display signals of puzzle cell map image with hands selection positions, and the wireless display glass projector project the puzzle cell image keys info on its lenses, so user can see which keys they select. The left side screen area is for left hand keys and the right side screen area is for the right hand keys. The lenses center area can optional to reserved for display robot text feedback, and real-time video image of user action.

9. Special Gesture Interface Robot has special unique fingers hand sign gestures enhance puzzle cell selection control accuracy. A special GIR gesture hand sign to moving fingers like spider walking its legs to change nearby puzzle cell selection. Moving like spider legs walking gesture, the puzzle cell row-column lines like spider web net. So tiny moving fingers in waking direction of up, down, left, right, the program can detect to change select key.

10. Special Gesture Interface Robot according to claim 9 has special unique fingers hand sign gestures enhance puzzle cell selection control accuracy. A special hold to grip click hand sign, special GIR hand gesture feature is useful when user need to rush to click a virtual key for sure in a emergency situation such as in spaceship out of control or user has shaking hands illness problem, the program will support for the need.

11. Special Gesture Interface Robot according to claim 9 has special unique fingers hand sign gestures enhance puzzle cell selection control accuracy. A special GIR hand sign gesture to continuous click without pull hand back to unlock, by moving like fish swimming its fins, moving fingers 100, 105, 106, 107, 108 up and down that cause hand palm center blinking, a vision tracking function detect perform continuous clicks, without pulling hand back to 1^st zone.

12. Special Gesture Interface Robot according to claim 9 has special unique fingers hand sign gestures enhance puzzle cell selection control accuracy. A special GIR hand sign look like a gun gesture point to puzzle cell, so the program see the hand holding and then fingers point out that make different, program select to lock the key. Tiny gun gesture point area make vision-tracking accuracy, so user move rotate finger gun point, small movement to change key selection.

16. Gesture Interface Robot enable advance gesture interface that TouchScreen Mouse combine puzzle cell virtual keyboard in sandwich layers method. It is a virtual touch screen that user can use hand gesture to move the mouse position and select mouse click action to operate computer.

17. Gesture Interface Robot according to claim 16 Gesture Interface Robot TouchScreen Mouse, For user to convince to move their hand control mouse point, the Right Hand TouchScreen Mouse program function can be setup to start cursor position will be in monitor LEFT-TOP corner position that is video card monitor 0,0 position. On the other hand, for the left hand will be determined position using X, Y value between center point of section I, II, III, and IV, then the LEFT Hand TouchScreen Mouse program function can setup to start cursor position in monitor Right-Bottom corner position, Gesture Interface Robot Puzzle Cell Keys can be define in software function coding by allow the keys to multiple click, multiple speed, different move distant, enable multiple clicks by 1 gesture action and also allow to control the lock or unlock the key that to enable re-click in 3^rd zone key. When user use GIR special gesture hand sign can continuous click virtual keys easily in 3^rd selection click zone.