Gesture interface robot

• US 9,696,813 B2
• Filed: 05/27/2015
• Issued: 07/04/2017
• Est. Priority Date: 05/27/2015
• Status: Active Grant

• Current Assignee: Hsien-Hsiang Chiu
• Original Assignee: Hsien-Hsiang Chiu
• Inventors: Chiu, Hsien-Hsiang
• Primary Examiner(s): VILLECCO, JOHN M
• Application Number:
  US14/723,435
• Publication Number:
  US 20160350589A1
• Time in Patent Office: 769 Days
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

A gesture interface robot (GIR) automatically measures a work zone to establish virtual puzzle cells located in a user gesture comfortable area. The GIR allows the user to easily move hands to press selected virtual keys so as to prevent injury. The GIR draws virtual keyboard images that display commands on virtual puzzle cells. The GIR uses video vision sensing to detect user'"'"'s hand location on the virtual keyboard images. Special unique gesture hand sings are designed for enhancing hand control virtual key selections. A push-hand-z-dimensional distant is divided into 3 selectable zones. A real-time display highlights a selected key graphic image to be visually seen by the user. Using a UIRT cable to send IR signal to a remote control computer. The GIR instantly generates a virtual puzzle cell keyboard or controller that the user selected. The GIR introduces a new gesture interface method, a touch screen mouse combined with puzzle cell virtual keys in sandwich layers gesture zones and a method for operating computer keys and mouse operations.

Claims

1. A gesture interface robot (GIR) comprising

a platform having wheels;

a main computer;

a video vision sensor module to sense gestures of a user;

a universal infrared receiver transmitter;

a micro controller board;

a puzzle cell map virtual keyboard control program (PCMVKCP);

a plurality of motor modules controllable by the micro controller board; and

one or more display monitors or projectors;

wherein the main computer, the video vision sensor module, the universal infrared receiver transmitter, the micro controller board, the plurality of motor modules and the one or more display monitors or projectors are attached to the platform;

wherein the PCMVKCP divides an three-dimensional virtual workspace zone into a puzzle cell row-column formation;

wherein the PCMVKCP divides the three-dimensional virtual workspace zone into a first, second and third selectable gate zones along a direction perpendicular to a surface of the puzzle cell row-column formation

wherein the first selectable gate zone is to unlock a selected key gate zone;

wherein the second selectable gate zone is to select a virtual key zone; and

wherein the third selectable gate zone is to click a selected zone.

2. The GIR of claim 1 further comprising a portable power source selected from the group consisting of a rechargeable battery, a solar cell, a fuel cell, a rotation generator, a wind turbine and a thermo electron generator.

3. The GIR of claim 1, wherein a puzzle cell virtual keyboard in the three-dimensional virtual workspace zone is presented to the user and wherein a motion of a hand and fingers of the user selecting a selected key of the puzzle cell virtual keyboard is captured by the video vision sensor module.

4. The GIR of claim 3 further comprising a plurality of pre-recorded macro scripts, wherein the GIR executes a selected pre-recorded macro script of the plurality of pre-recorded macro scripts corresponding to the selected key of the puzzle cell virtual keyboard to send infrared signals, Bluetooth signals or WiFi signals to remotely control a computer, a machine, an intelligent robot, an auto self driving car, a flight jet, a spaceship, a radio, a disco light, a Morse keyboard, a telescope, a television, a digital video disk, a mouse or an internet browser.

5. The GIR of claim 1, wherein the main computer includes a web server function so as to control a computer, a machine, an intelligent robot, an auto self driving car, a flight jet, a spaceship, a radio, a disco light, a Morse keyboard, a telescope, a television, a digital video disk, a mouse or an internet browser through an automation program.

6. The GIR of claim 1, wherein the PCMVKCP determines a center, a width and a length of the three-dimensional virtual workspace zone and the PCMVKCP activates a selected motor module of the plurality of motor modules to rotate the video vision sensor module when the user walks out of a video viewable area.

7. The GIR of claim 1, wherein the GIR defines a hand gesture to move fingers of the user to change a selection of a selected puzzle cell.

8. The GIR of claim 7, wherein the defined hand gesture improves a control accuracy of the selection of the selected puzzle cell.

9. The GIR of claim 1, wherein a virtual puzzle cell map keys control panel graphic image is drawn on the one or more display monitors or projectors;

wherein the virtual puzzle cell map keys control panel graphic image is divided into a left, a center and a right zones and wherein a real time video image of the user is displayed in the center zone of the virtual puzzle cell map keys control panel graphic image.

10. The GIR of claim 1, wherein the GIR accepts voice commands to rotate a selected motor module of the plurality of motor modules.

11. The GIR of claim 1, wherein the GIR displays a virtual touch screen.

12. The GIR of claim 1 further comprising a wireless key-indication device configured to be worn on a palm, an arm or a body of the user.

13. The GIR of claim 1, wherein the GIR is of a human shape;

wherein the GIR moves from a first location to a second location by the wheels; and

wherein a surface of the one or more display monitors or projectors is changed from a first predetermined direction to a second predetermined direction.

14. The GIR of claim 1 further comprising a wireless display glass projector to project key information of a puzzle cell image on lenses.

15. The GIR of claim 1 further comprising a visual image projector;

wherein the GIR moves the one or more display monitors or projectors or the visual image projector along a predetermined direction; and

wherein the one or more display monitors or projectors or the visual image projector displays letters a to z, numbers 0 to 9, symbols, function keys, open computer programs, Media, a running DVD player, a playing music, a video, an internet browser, playing games, or computer functions.

16. The GIR of claim 1, wherein the GIR sends an enter key command directly to an activation program.

17. The GIR of claim 1, wherein the GIR activates a virtual touch screen mouse function and wherein the GIR disables a right-hand selection function and enables a left-hand selection function.

18. The GIR of claim 1, wherein the GIR calculates a ratio of X, Y distance between virtual center points and applies the ratio to move a mouse cursor position.

19. The GIR of claim 1, wherein the GIR recognizes a gesture of pushing out a hand of the user as to perform a click action.

20. The GIR of claim 1 further comprising at least one network equipment.

21. The GIR of claim 1 further comprising a speech recognition program function and an array of microphones.

22. The GIR of claim 1, wherein the micro controller board includes sensor modules and the micro controller board sends signals to the main computer.

23. The GIR of claim 1, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions.

24. A gesture interface robot (GIR) comprising

a main computer;

a video vision sensor module to sense gestures of a user;

one or more display monitors or projectors; and

a puzzle cell map virtual keyboard control program (PCMVKCP);

wherein the PCMVKCP divides a three-dimensional virtual workspace zone into a puzzle cell row-column formation;

wherein the PCMVKCP divides a space of the three-dimensional virtual workspace zone into selectable sandwich layer work zones along a direction perpendicular to a surface of the puzzle cell row-column formation; and

wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions.

25. The GIR of claim 24, wherein the GIR allows automatic measurement of the three-dimensional virtual workspace zone of the user, assigns a virtual center point, creates the three-dimensional virtual workspace zone in a conformable area, establishes puzzle cell mapping keys, and presents virtually controlling panel in front of the user to be clicked.

26. The GIR of claim 24, wherein the video vision sensor module is configured to detect three-dimensional positional values of joints of the user and the GIR provides real-time tracking of locations of hands of the user.

27. The GIR of claim 24, wherein the GIR uses a relative distant on hand locations to a center point to determine X, Y position of a puzzle cell position and wherein the GIR recognizes user push hand locations in Z direction.

28. The GIR of claim 24, wherein the PCMVKCP divides the three-dimensional virtual workspace zone into a first, second and third selectable gate zones along a direction perpendicular to a surface of the puzzle cell row-column formation

wherein the first selectable gate zone is to unlock a selected key gate zone;

wherein the second selectable gate zone is to select a virtual key zone; and

wherein the third selectable gate zone is to click a selected zone.

29. The GIR of claim 24, wherein the GIR divides the three-dimensional virtual workspace zone into four sections including section I of (X+, Y+), section II of (X−

, Y+), section III of (X+, Y−

), and section IV of (X−

, Y−

) and wherein the section I and the section III are operable by a right hand of the user and wherein the section II and the section IV are operable by a left hand of the user.

30. The GIR of claim 24, wherein the GIR detects a hand of the user moving up, down, left, right, push and pull.

31. The GIR of claim 24, wherein the GIR calculates a ratio of X, Y distance between virtual center points and applies the ratio to move a mouse cursor position.

32. The GIR of claim 24, wherein an initial cursor position is at an upper left corner or at a lower right corner of a virtual touch screen.

33. The GIR of claim 24, wherein the GIR recognizes a gesture of pushing out a hand of the user as to perform a click action.

34. The GIR of claim 24, wherein the GIR recognizes a gesture to control a lock or an unlock action.

35. The GIR of claim 24, wherein the GIR defines a gesture enhancing selection control accuracy.

36. The GIR of claim 24, wherein the GIR defines a hand gesture for a touchscreen mouse grip function.

37. The GIR of claim 24, wherein the touchscreen mouse is combined with a virtual puzzle cell keys control panel using the sandwich layers functions.

38. The GIR of claim 24, wherein the GIR supports a gesture action of the touchscreen mouse combining the sandwich layers functions with virtual control panel keys zone functions.

39. The GIR of claim 24, wherein each hand and fingers gestures and position values on each video frame is compared and distinguished by a respective hand sign on a puzzle cell area.

40. The GIR of claim 24, wherein the GIR tracks X, Y values of hands of the user and tracks Z values between a body of the user and the hands of the user.

41. The GIR of claim 24, wherein the GIR activates a virtual touch screen mouse function and wherein the GIR disables a right-hand selection function and enables a left-hand selection function.

42. The GIR of claim 24, wherein the GIR supports a hand sign language function.

43. A gesture interface robot (GIR) comprising

a main computer;

a video vision sensor module to sense gestures of a user;

one or more display monitors or projectors; and

a puzzle cell map virtual keyboard control program (PCMVKCP);

wherein the PCMVKCP divides a three-dimensional virtual workspace zone into a puzzle cell row-column formation;

wherein the PCMVKCP divides a space of the three-dimensional virtual workspace zone into a first, second and third selectable gate zones along a direction perpendicular to a surface of the puzzle cell row-column formation;

wherein the video vision sensor module includes a vision sensor, an RGB video camera, a web camera or a video sensor camera equipped with an infrared emitter and an infrared signal reflection detect sensor;

wherein the first selectable gate zone is to unlock a selected key gate zone;

wherein the second selectable gate zone is to select a virtual key zone; and

wherein the third selectable gate zone is to click a selected zone.

44. The GIR of claim 43, wherein the GIR allows automatic measurement of the three-dimensional virtual workspace zone of the user, assigns a virtual center point, creates the three-dimensional virtual workspace zone in a conformable area, establishes puzzle cell mapping keys, and presents virtually controlling panel in front of the user to be clicked.

45. The GIR of claim 44, wherein said creating the three-dimensional virtual workspace zone in the conformable area is to establish a three-dimensional workspace, establish a width of the three-dimensional virtual workspace zone being a pre-determined times of a sum of two shoulder widths of the user and to establish a length of the three-dimensional virtual workspace zone being a function of a length of an arm of the user and wherein a direction of the length is along an arm pushing out direction of the user.

46. The GIR of claim 43, wherein a puzzle cell size is calculated by a workspace zone size divided by a number of rows and a number of columns.

47. The GIR of claim 43, wherein the GIR divides the three-dimensional virtual workspace zone into four sections including section I of (X+, Y+), section II of (X−

, Y+), section III of (X+, Y−

), and section IV of (X−

, Y−

).

48. The GIR of claim 43, wherein the GIR detects a hand of the user moving up, down, left and right, push and pull.

49. The GIR of claim 43, wherein the GIR uses a puzzle cell mapping gesture command method and the user extends a user'"'"'s hand forward to confirm a selection.

50. The GIR of claim 43, wherein the GIR uses a relative distant on hand locations to a center point to determine X, Y position of a puzzle cell position and wherein the GIR recognizes user push hand locations in Z direction.

51. The GIR of claim 43, wherein the puzzle cell row-column formation includes a keyboard.

52. The GIR of claim 43, wherein the video vision sensor module is configured to detect three-dimensional positional values of joints of the user and the GIR provides real-time tracking of locations of hands of the user.

53. The GIR of claim 52, wherein a virtual puzzle cell map keys control panel graphic image is drawn on the one or more display monitors or projectors;

wherein the virtual puzzle cell map keys control panel graphic image is divided into a left, a center and a right zones and wherein a real time video image of the user is displayed in the center zone of the virtual puzzle cell map keys control panel graphic image.

54. The GIR of claim 43, wherein values read by the video vision sensor module are available for the PCMVKCP to use them.

55. The GIR of claim 43, wherein the GIR detects motions of fingers of the user.

56. The GIR of claim 43, wherein the GIR defines a gesture enhancing selection control accuracy.

57. The GIR of claim 43, wherein the GIR defines a hand gesture to move fingers of the user to change a selection of a selected puzzle cell.

58. The GIR of claim 43, wherein the GIR detects a continuously click virtual key without pull a hand of the user back.

59. The GIR of claim 43, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions.

60. The GIR of claim 43, wherein the GIR recognizes a gesture of pushing out a hand of the user as to perform a click action.

61. The GIR of claim 43, wherein the GIR determines a first ratio of a first width to a first height of a video view frame;

wherein the GIR compares the first ratio with a second ratio of a second width to a second height of a monitor screen; and

the GIR moves a mouse cursor by a distance based on the comparison of the first ratio with the second ratio.

62. The GIR of claim 43, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions and wherein the puzzle cell row-column formation includes a keyboard.

63. The GIR of claim 43, wherein the puzzle cell row-column formation includes a keyboard and wherein the GIR supports a hand sign language function.

64. The GIR of claim 43, wherein each hand and fingers gestures and position values on each video frame is compared and distinguished by a respective hand sign on a puzzle cell area.

65. The GIR of claim 43, wherein the GIR supports a hand sign language function.

66. The GIR of claim 43, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions and wherein the GIR supports a hand sign language function.

67. The GIR of claim 43, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions;

wherein the GIR supports a hand sign language function; and

wherein the puzzle cell row-column formation includes a keyboard.

68. The GIR of claim 43, wherein the GIR recognizes a pointer stick configured to be held by a mouth of the user.

69. The GIR of claim 43, wherein the GIR sends an enter key command directly to an activation program.

70. The GIR of claim 43, further comprising a plurality of pre-recorded macro scripts, wherein the GIR executes a selected pre-recorded macro script of the plurality of pre-recorded macro scripts corresponding to the selected key of the puzzle cell virtual keyboard to send infrared signals, Bluetooth signals or WiFi signals to remotely control a computer, a machine, an intelligent robot, an auto self driving car, a flight jet, a spaceship, a radio, a disco light, a morse keyboard, a telescope, a television, a digital video disk, a mouse or an internet browser.

71. The GIR of claim 43, wherein the main computer includes a web server function so as to control a computer, a machine, an intelligent robot, an auto self driving car, a flight jet, a spaceship, a radio, a disco light, a morse keyboard, a telescope, a television, a digital video disk, a mouse or an internet browser through an automation program.

72. The GIR of claim 43 further comprising a universal infrared receiver transmitter.

73. The GIR of claim 43 further comprising at least one network equipment.

74. The GIR of claim 43, wherein the one or more display monitors or projectors or a visual image projector displays letters a to z, numbers 0 to 9, symbols, function keys, open computer programs, Media, a running DVD player, a playing music, a video, an internet browser, playing games, or computer functions.

75. The GIR of claim 43 further comprising a speech recognition program function and an array of microphones.

76. The GIR of claim 43, wherein the GIR is equipped with voice speaking program function.

77. The GIR of claim 43 further comprising a wireless key-indication device configured to be worn on a palm, an arm or a body of the user.

78. The GIR of claim 43 further comprising a wireless display glass.

79. The GIR of claim 43 further comprising a micro controller board, wherein the micro controller board includes sensor modules and the micro controller board sends signals to the main computer.

80. The GIR of claim 43, wherein the GIR includes a virtual mouse, a keyboard, control panels, a computer interface method, a touchscreen mouse, virtual puzzle cell mouse keys, a virtual control panel keyboard, a mouse key controller or a hand sign language function.

81. The GIR of claim 43, wherein the GIR uses a computer interface method to remotely control a computer, a machine, an intelligent robot, an auto self driving car, a flight jet, a spaceship, a radio, a disco light, a morse keyboard, a telescope, a television, a digital video disk, a mouse or an internet browser.

82. The GIR of claim 43 further comprising a micro controller board, wherein the micro controller board includes sensor modules and the micro controller board sends signals to the main computer and wherein controls rotations of a plurality of motors through the micro controller board.

83. The GIR of claim 43, wherein the GIR tracks X, Y values of hands of the user and tracks Z values between a body of the user and the hands of the user.

84. A gesture interface robot (GIR) comprising

a platform having wheels;

a main computer;

a video vision sensor module to sense gestures of a user;

a universal infrared receiver transmitter;

a micro controller board;

a wireless display glass projector to project key information of a puzzle cell image on lenses;

a plurality of motor modules controllable by the micro controller board; and

a display monitor;

wherein the main computer, the video vision sensor module, the universal infrared receiver transmitter, the micro controller board, the plurality of motor modules and the display monitor are attached to the platform.

85. The GIR of claim 84 further comprising a three-dimensional virtual workspace zone, wherein a puzzle cell virtual keyboard in the three-dimensional virtual workspace zone is presented to the user and wherein a motion of a hand and fingers of the user selecting a selected key of the puzzle cell virtual keyboard is captured by the video vision sensor module.

86. The GIR of claim 84, wherein the GIR includes a virtual mouse, a keyboard, control panels, a computer interface method, a touchscreen mouse, virtual puzzle cell mouse keys, a virtual control panel keyboard, a mouse key controller or a hand sign language function.

87. The GIR of claim 84, wherein the GIR further comprises a puzzle cell map virtual keyboard control program (PCMVKCP);

wherein the PCMVKCP divides an virtual workspace zone into a puzzle cell row-column formation;

wherein the PCMVKCP divides the virtual workspace zone into a first, second and third selectable gate zones along a direction perpendicular to a surface of the puzzle cell row-column formation

wherein the first selectable gate zone is to unlock a selected key gate zone;

wherein the second selectable gate zone is to select a virtual key zone; and

wherein the third selectable gate zone is to click a selected zone.

88. The GIR of claim 84 further comprising a three-dimensional virtual workspace zone, wherein the three-dimensional virtual workspace zone includes a touchscreen mouse having sandwich layers functions.