Natural human to robot remote control

• US 9,079,313 B2
• Filed: 03/15/2011
• Issued: 07/14/2015
• Est. Priority Date: 03/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Olivier, Charles F. III, Fouillade, Jean Sebastien
• Primary Examiner(s): Tran, Dalena
• Assistant Examiner(s): Figueroa, Jaime
• Application Number:
  US13/048,123
• Publication Number:
  US 20120239196A1
• Time in Patent Office: 1,582 Days
• Field of Search: 700/245, 700/259, 901/1, 901/2, 901/47, 382/103, 382/107, 382/181, 382/259, 318/568.12, 318/568.16
• US Class Current: 1/1
• CPC Class Codes: B25J 11/0015 Face robots, animated artificial faces for imitating human expressions
• CPC Class Codes: B25J 13/02 Hand grip control means handles or pedals for crane control B66C13/56; for measuring the force applied to control members G01L5/22; hand-held casings for switching devices, e.g. joy-sticks H01H9/0214
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/40174 Robot teleoperation through internet
• CPC Class Codes: G05B 2219/40176 Encode operator actions into symbolic commands for transmission to remote
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

The subject disclosure is directed towards controlling a robot based upon sensing a user'"'"'s natural and intuitive movements and expressions. User movements and/or facial expressions are captured by an image and depth camera, resulting in skeletal data and/or image data that is used to control a robot'"'"'s operation, e.g., in a real time, remote (e.g., over the Internet) telepresence session. Robot components that may be controlled include robot “expressions” (e.g., audiovisual data output by the robot), robot head movements, robot mobility drive operations (e.g., to propel and/or turn the robot), and robot manipulator operations, e.g., an arm-like mechanism and/or hand-like mechanism.

Claims

1. In a computing environment, a system comprising:

a control program implemented in a remote device relative to a robotic device and configured to receive image data and depth data captured at a location remote from the robotic device, convert the image data and the depth data into skeletal data, process the skeletal data into one or more action commands that when processed by the robotic device operate one or more components of the robotic device, and transmit the one or more action commands to the robotic device over a connection to control the robotic device.

2. The system of claim 1 further comprising:

a communications link by which the one or more action commands are communicated to the robotic device over the connection from a network associated with the remote device to a different network associated with the robotic device.

3. The system of claim 2 wherein the communications link comprises an Internet connection.

4. The system of claim 1 wherein the one or more components of the robotic device comprise a mobility drive, and wherein the one or more action commands control the mobility drive to propel the robot or turn the robot, or both propel the robot and turn the robot.

5. The system of claim 1 wherein the one or more components of the robotic device comprise a robot head, and wherein the one or more action commands control the vertical or horizontal angle, or both the vertical and horizontal angle, of the robot head.

6. The system of claim 1 wherein the one or more components of the robotic device comprise a manipulator, and wherein the one or more action commands control the operation of the manipulator.

7. The system of claim 1 wherein the one or more components of the robotic device comprise a robot hand-like mechanism, and wherein the one or more action commands control the operation of the robot hand-like mechanism.

8. The system of claim 1 wherein the one or more components of the robotic device comprise an expression output mechanism, wherein the image data and the depth data include facial expression data, wherein the control program is further configured to process the facial expression data into an expression type that controls the expression output mechanism of the robotic device.

9. In a computing environment, a method performed at least in part on at least one processor, comprising:

receiving image data and depth data;

identifying head movement of a user via the image data and the depth data;

detecting changes to the head movement of the user; and

transmitting data corresponding to the detected changes, including transmitting data representing the head movement, to a robot that uses the data corresponding to the changes detected to control operation of one or more components of the robot to match head movement of the robot with the head movement of the user.

10. The method of claim 9 further comprising:

receiving visual data from a point-of-view of the robot responsive to the head movement of the robot.

11. The method of claim 9 further comprising:

identifying skeletal data of the user via the image data and the depth data;

detecting changes to the skeletal data corresponding to arm movement of the user; and

transmitting skeletal change data corresponding to the changes to the skeletal data to the robot to operate a robot manipulator based on the skeletal change data.

12. The method of claim 9 further comprising:

identifying skeletal data of the user via the image data and the depth data;

detecting changes to the skeletal data corresponding to hand movement of the user; and

transmitting skeletal change data corresponding to the changes to the skeletal data to the robot to operate a robot hand-like mechanism based on the skeletal change data.

13. The method of claim 9 further comprising:

identifying expression data of the user via the image data and the depth data;

detecting changes to the expression data, including determining an expression type; and

transmitting data corresponding to the determined expression type to the robot to operate a robotic expression system to output a robotic expression on the robot based on the data corresponding to the determined expression type.

14. The method of claim 9 further comprising:

identifying skeletal data of the user via the image data and the depth data;

detecting changes to the skeletal data corresponding to walking or turning movements of the user; and

transmitting skeletal change data corresponding to the changes to the skeletal data to the robot to operate a robot mobility drive mechanism to propel or turn the robot, or both propel and turn the robot, based on the skeletal change data.

15. The method of claim 9 wherein transmitting the data occurs in real time during a remote telepresence session to control operation of the robot, and further comprising:

receiving sensory information corresponding to the operation of the robot during the remote telepresence session transmitted from one or more sensors coupled to the robot.

16. One or more computer storage devices having computer-executable instructions, which in response to execution by a computer, cause the computer to perform steps comprising:

receiving information at a robot corresponding to skeletal data converted from image data and depth data, the image data and the depth data captured from a user via a control device remote from the robot, the control device including user interaction technology and configured to convert the image data and the depth data to the skeletal data;

processing the information to control operation of one or more components of the robot; and

transmitting other information corresponding to the operation of the one or more components of the robot to the control device, the other information including sensory data captured by one or more sensors coupled to the robot as the one or more components of the robot are controlled using the received information.

17. The one or more computer storage devices of claim 16 having further computer-executable instructions comprising:

receiving information at the robot corresponding to facial expression data converted from the image data and the depth data; and

processing the information to control operation of an expression system of the robot.

18. The one or more computer storage devices of claim 16 wherein the information comprises one or more action commands or data, or both action commands and data, computed at the control device based upon the skeletal data.

19. The one or more computer storage devices of claim 16 wherein processing the information to control the operation of one or more components of the robot comprises at least one of controlling robot head movements, controlling robot manipulator movements, or controlling a robot mobility drive.

20. The one or more computer storage devices of claim 19 having further computer-executable instructions comprising:

receiving information at the robot corresponding to facial expression data converted from the image data and the depth data; and

processing the information to control operation of an expression system of the robot.