Remote presence robotic system

• US 9,931,749 B2
• Filed: 04/14/2016
• Issued: 04/03/2018
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Daleroy Sibanda, Emre Tanirgan, John C. Nappo, Peter A. Zachares
• Original Assignee: Daleroy Sibanda, Emre Tanirgan, John C. Nappo, Peter A. Zachares
• Inventors: Nappo, John C., Sibanda, Daleroy, Tanirgan, Emre, Zachares, Peter A.
• Primary Examiner(s): Nguyen, Bao Long T
• Application Number:
  US15/099,314
• Publication Number:
  US 20160303735A1
• Time in Patent Office: 719 Days
• Field of Search: 700257, 700250, 700253, 700258, 901 1, 31856816, 31856825, 701 2
• CPC Class Codes: B25J 13/00 Controls for manipulators programme controls B25J9/16; control in general G05
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: B25J 9/161 Hardware, e.g. neural networks, fuzzy logic, interfaces, processor
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G05B 2219/35482 Eyephone, head-mounted 2-D or 3-D display, also voice and other control
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A remote presence system includes a robot, a headset, and a computer system. The robot includes a camera mounted on a motorized manipulator and a control system. The headset includes a head position sensor and a display. The computer system configured to perform operations including: receiving head movement data indicative of head movements of a user wearing the headset from the head position sensor; causing, using the head movement data, the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user; and establishing a video feed from the camera to the display.

Claims

1. A system comprising:

a robot comprising;

a motorized manipulator;

a camera mounted on the motorized manipulator;

a camera position sensor; and

a control system configured to cause the motorized manipulator to move the camera;

a headset comprising;

a head position and orientation sensor; and

a display; and

a computer system configured to perform operations comprising;

receiving head movement data indicative of head movements of a user wearing the headset from the head position and orientation sensor;

causing, using the head movement data, the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user, wherein causing the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user comprises supplying camera movement data from the camera position sensor and the head movement data to a closed-loop control module; and

establishing a video feed from the camera to the display.

2. The system of claim 1, wherein:

the robot comprises first and second cameras mounted side by side on the motorized manipulator;

the display comprises first and second display screens, one for each eye of the user; and

establishing the video feed comprises establishing a first video feed from the first camera to the first display screen and establishing a second video feed from the second camera to the second display screen, thereby creating a stereoscopic display experience for the user.

3. The system of claim 1 comprising a user input device, wherein the robot comprises a drive system, and wherein the operations of the computer system comprise:

receiving input from the user input device; and

causing the control system of the robot to move the robot using the drive system based on the input from the user input device.

4. The system of claim 3, wherein the drive system comprises a differential drive system comprising at least two wheels and a respective motor for each wheel.

5. The system of claim 1, wherein the closed-loop control module comprises a low-pass filter and a high-pass filter configured to regulate the speed and frequency of movement of the motorized manipulator.

6. The system of claim 1, wherein the closed-loop control module comprises a head Kalman filter configured to filter the head movement data and a camera Kalman filter configured to file the camera movement data.

7. The system of claim 1, wherein the head position sensor and the camera position sensor each comprises a three-axis accelerometer and a three-axis gyroscope.

8. The system of claim 1, wherein the motorized manipulator comprises a plurality of rotating motors configured to control roll, pitch, and yaw of the camera.

9. The system of claim 1, wherein the motorized manipulator comprises a plurality of linear actuators configured to move the camera both vertically and within a horizontal plane.

10. The system of claim 1, wherein the robot comprises a microphone, and wherein the headset comprises one or more speakers, and wherein the operations of the computer system comprise establishing an audio feed from the microphone to the one or more speakers.

11. The system of claim 1, wherein causing the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user comprises sending instructions based on the head movement data to the control system over a data communications network comprising at least one wireless communications link.

12. The system of claim 1, wherein causing the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user comprises causing the camera to mimic the head movements of the user in real-time or near real-time.

13. The system of claim 1, wherein the operations of the computer system comprise maintaining a reservation calendar for the robot and providing a user interface for the reservation calendar configured to allow a plurality of users to reserve blocks of time during which the users can control the robot.

14. A method comprising:

receiving, by a computer system, head movement data indicative of head movements of a user wearing a headset from a head position sensor on the headset;

causing, by the computer system using the head movement data, a control system of a robot to move a motorized manipulator of the robot so that a camera mounted on the motorized manipulator mimics the head movements of the user, wherein causing the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user comprises supplying camera movement data from a camera position sensor of the robot and the head movement data to a closed-loop control module; and

establishing, by the computer system, a video feed from the camera to a display of the headset.

15. A non-transitory computer readable medium having stored thereon executable instructions that when executed by one or more processors of a computer control the computer to perform steps comprising:

receiving head movement data indicative of head movements of a user wearing a headset from a head position sensor on the headset;

causing, using the head movement data, a control system of a robot to move a motorized manipulator of the robot so that a camera mounted on the motorized manipulator mimics the head movements of the user, wherein causing the control system of the robot to move the motorized manipulator so that the camera mimics the head movements of the user comprises supplying camera movement data from a camera position sensor of the robot and the head movement data to a closed-loop control module; and

establishing a video feed from the camera to a display of the headset.