Gesture enabled telepresence robot and system

• US 9,552,056 B1
• Filed: 08/15/2012
• Issued: 01/24/2017
• Est. Priority Date: 08/27/2011
• Status: Expired due to Fees

• Current Assignee: Fellow Incorporated
• Original Assignee: Fellow Robots, Inc.
• Inventors: Fan, Zhengqin, Hardie, Alaina L., Barry, Daniel T.
• Primary Examiner(s): Black, Thomas G
• Assistant Examiner(s): THOMAS, ANA D
• Application Number:
  US13/586,732
• Time in Patent Office: 1,623 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/174 Facial expression recognition

Abstract

A telepresence robot includes a mobile platform, a camera, and a computer in communication with the platform and the camera. The computer receives video information from a remotely located camera relating to a gesture made by a remote human operator, and the robot is configured to operate based on the gesture. The computer may further predict an emotional state of a person interacting with the robot, based on video information received from the robot camera. A telepresence robot system includes: a telepresence robot having a mobile platform, a robot camera, a robot display, and a robot computer; and a control station, located remote to the robot, having a control station camera, a control station display, and a control station computer.

Claims

1. A telepresence robot comprising:

a mobile platform;

a first robot camera; and

a computer in communication with the platform and the first robot camera, said computer having a processor and computer-readable memory,

wherein said computer is configured to receive video information from at least one remotely located camera and sensed 3D information from at least one remotely located 3D sensor, the video information and sensed 3D information signifying a physical body gesture made by a remotely-located human operator, and wherein said telepresence robot is configured to perform an imitating action of the physical body gesture made by the remotely-located human operator, and

wherein said computer is further configured to receive video information from the first robot camera comprising a video image of a person interacting with the telepresence robot and to predict an emotional state of the person based on the received video information.

2. The telepresence robot of claim 1, further comprising an infrared camera and/or biometric sensor, wherein said computer is further configured to receive information from the infrared camera and/or biometric sensor relating to the person and to predict an emotional state of the person based on the information received from the first robot camera and the infrared camera.

3. The telepresence robot of claim 2, further comprising a retinal scanner configured to scan the eyes of the person interacting with the telepresence robot, wherein said computer is further configured to receive information from the retinal scanner and to predict an emotional state of the person based on the information received from the first robot camera, the infrared camera and the retinal scanner.

4. The telepresence robot of claim 1, further comprising:

a robot display;

a robot microphone; and

a robot speaker,

wherein said computer is further in communication with the display, microphone and speaker.

5. The telepresence robot of claim 4, further comprising a second robot camera, wherein said first robot camera is disposed on a pan and tilt base, and the second robot camera is provided in a fixed position with a downward view.

6. The telepresence robot of claim 4, wherein the display is a touchscreen display.

7. The telepresence robot of claim 4, wherein said computer is configured to receive video information from the at least one remotely located camera and audio information from at least one remotely located microphone, and to display the received video information on the robot display, project the received audio information from the speaker, and to transmit information received from the first robot camera and the robot microphone to a remotely located computer.

8. The telepresence robot of claim 7, wherein said computer is further configured to receive an email address and a name of an operator of the remotely located computer, and to display the received email and name on the robot display.

9. The telepresence robot of claim 1, further comprising a 3D sensor.

10. The telepresence robot of claim 9, wherein the 3D sensor comprises at least one of a 3D video camera, a laser rangefinder, a 3D scanner, a structured light scanner, a stereo camera, an ultrasound sensor, a radar detector, a Doppler radar sensor, and a lidar detector.

11. The telepresence robot of claim 9, further comprising at least one of a tactile sensor, a cliff sensor, a bump sensor, a humidity sensor, a chemical sensor, a vibration sensor, an ultraviolet sensor, an ultrasonic sensor, and a temperature sensor.

12. The telepresence robot of claim 1, further comprising a robot location detector.

13. The telepresence robot of claim 12, wherein said computer-readable memory is configured to store scheduling information for the telepresence robot, said scheduling information including time and location information, wherein said computer is configured to access said scheduling information and to cause the telepresence robot to navigate to a specified location at a specified time based on said scheduling information.

14. A telepresence robot system comprising:

a telepresence robot having a mobile platform, a robot camera, a robot display, and a robot computer in communication with the platform and the robot camera, said robot computer having a processor and computer-readable memory; and

a control station located remote to the telepresence robot, said control station having a control station camera, a control station 3D sensor, a control station display, and a control station computer in communication with the control station camera and control station display, said control station computer having a processor and computer-readable memory;

wherein said control station camera and said control station 3D sensor are configured to detect video information and 3D sensed information signifying physical body gesture made by a human operator, and wherein said telepresence robot is configured to perform an imitating action of the detected physical body gesture, and

wherein said control station computer is further configured to receive video and/or biometric information from the control station camera and/or a biometric sensor relating to the human operator and to predict an emotional state of the operator based on the received information, and wherein the video and/or biometric information used to predict the emotional state of the operator comprises at least one of the following;

pupillary dilation, retinal patterns, blood flow, body fluid distribution, respiration, nostril movement, heart rate, and skin brightness.

15. The telepresence system of claim 14, wherein said control station computer is configured to receive video information from the control station camera relating to a gesture made by the human operator, and the control station processor causes the telepresence robot to operate based on said detected gesture.

16. The telepresence system of claim 14, wherein said robot computer is configured to receive video information from the control station camera relating to a gesture made by the human operator, and the robot processor causes the telepresence robot to operate based on said detected gesture.

17. The telepresence system of claim 14, wherein said control station further comprises an infrared camera, wherein said control station computer is further configured to receive information from the infrared camera relating to the human operator and to predict an emotional state of the operator based on the information received from the control station camera and the infrared camera.

18. The telepresence system of claim 14, wherein the control station display is a touchscreen display.

19. The telepresence system of claim 14, wherein said control station further comprises a 3D sensor.

20. The telepresence system of claim 19, wherein the 3D sensor comprises at least one of a 3D video camera, a laser rangefinder, a 3D scanner, a structured light scanner, a stereo camera, an ultrasound sensor, a radar detector, a Doppler radar sensor, and a lidar detector.

21. The telepresence system of claim 14, wherein said control station computer is configured to receive video information from a remotely located robot camera and audio information from a remotely located robot microphone, and to display the received video information on the control station display, project the received audio information from a control station speaker, and to transmit information received from the control station camera and a control station microphone to a remotely located robot computer.

22. The telepresence system of claim 21, wherein said control station computer is further configured to receive an email address and a name of a person interacting with the remotely located robot, and to display the received email and name on the control station display.

23. A telepresence robotic system comprising:

a telepresence robot having a mobile platform, a robot camera, a robot display, and a robot computer in communication with the platform and the robot camera, said robot computer having a processor and computer-readable memory; and

a control station located remote to the telepresence robot, said control station having a control station camera, a control station display, and a control station computer in communication with the control station camera and control station display, said control station computer having a processor and computer-readable memory;

wherein said control station computer receives video information from the robot camera and displays the received video information as a video image on the control station display, wherein said control station computer is configured to receive an input signal relating to a selection of a particular location in the video image on the control station display, and wherein said telepresence robot is configured to navigate to a particular location based on said selection.