Natural Human to Robot Remote Control

• US 20120239196A1
• Filed: 03/15/2011
• Published: 09/20/2012
• Est. Priority Date: 03/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Olivier, Charles F. III, Fouillade, Jean Sebastien
• Granted Patent: US 9,079,313 B2
• Time in Patent Office: Days
• US Class Current: 700/259
• CPC Class Codes: B25J 11/0015 Face robots, animated artificial faces for imitating human expressions
• CPC Class Codes: B25J 13/02 Hand grip control means handles or pedals for crane control B66C13/56; for measuring the force applied to control members G01L5/22; hand-held casings for switching devices, e.g. joy-sticks H01H9/0214
• CPC Class Codes: B25J 3/04 involving servo mechanisms servo-actuated heads B25J15/02
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: G05B 2219/40174 Robot teleoperation through internet
• CPC Class Codes: G05B 2219/40176 Encode operator actions into symbolic commands for transmission to remote
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

The subject disclosure is directed towards controlling a robot based upon sensing a user'"'"'s natural and intuitive movements and expressions. User movements and/or facial expressions are captured by an image and depth camera, resulting in skeletal data and/or image data that is used to control a robot'"'"'s operation, e.g., in a real time, remote (e.g., over the Internet) telepresence session. Robot components that may be controlled include robot “expressions” (e.g., audiovisual data output by the robot), robot head movements, robot mobility drive operations (e.g., to propel and/or turn the robot), and robot manipulator operations, e.g., an arm-like mechanism and/or hand-like mechanism.

Claims

1. In a computing environment, a system comprising, a camera configured to capture image and depth data from which provide skeletal data is produced, a control program configured to process the skeletal data into one or more action commands that when processed by a robotic device, operate one or more components of the robotic device.

2. The system of claim 1 wherein the control program is implemented in a remote device relative to the robotic device, and further comprising a communications link by which the action commands are communicated to the robotic device.

3. The system of claim 2 wherein the communications link comprises an Internet connection.

4. The system of claim 1 wherein the one or more components of the robotic device comprise a mobility drive, and wherein the one or more action commands control the mobility drive to propel the robot or turn the robot, or both propel the robot and turn the robot.

5. The system of claim 1 wherein the one or more components of the robotic device comprise a robot head, and wherein the one or more action commands control the vertical or horizontal angle, or both the vertical and horizontal angle, of the robot head.

6. The system of claim 1 wherein the one or more components of the robotic device comprise a manipulator, and wherein the one or more action commands control the operation of the manipulator.

7. The system of claim 1 wherein the one or more components of the robotic device comprise a robot hand-like mechanism, and wherein the one or more action commands control the operation of the robot hand-like mechanism.

8. The system of claim 1 wherein the one or more components of the robotic device comprise an expression output mechanism, wherein the image and depth data include facial expression data, wherein the control program is further configured to process the facial expression data into an expression type that controls the expression output mechanism of the robot.

9. In a computing environment, a method performed at least in part on at least one processor, comprising:

capturing facial expressions of a user via an image and depth camera;

detecting changes to the user'"'"'s facial expressions; and

transmitting data corresponding to the changes to the user'"'"'s facial expressions to a robot that uses the data to control the operation of a robotic expression system.

10. The method of claim 9 wherein detecting the changes includes determining an expression type, and wherein transmitting the data comprises transmitting the expression type.

11. The method of claim 9 further comprising capturing skeletal data of a user via the image and depth camera, detecting changes to the skeletal data corresponding to user arm movement, and transmitting other data corresponding to the changes to the skeletal data to the robot to operate a robot manipulator based on the other data.

12. The method of claim 9 further comprising capturing skeletal data of a user via the image and depth camera, detecting changes to the skeletal data corresponding to user hand movement, and transmitting other data corresponding to the changes to the skeletal data to the robot to operate a robot hand-like mechanism based on the other data.

13. The method of claim 9 further comprising capturing skeletal data of a user via the image and depth camera, detecting changes to the skeletal data corresponding to user head movement, and transmitting other data corresponding to the changes to the skeletal data to the robot to operate a robot head or robot camera, or both robot head and robot camera, based on the other data.

14. The method of claim 9 further comprising capturing skeletal data of a user via the image and depth camera, detecting changes to the skeletal data corresponding to user walking or turning movements, and transmitting other data corresponding to the changes to the skeletal data to the robot to operate a robot mobility drive mechanism to propel or turn the robot, or both propel and turn the robot, based on the other data.

15. The method of claim 9 wherein transmitting the data occurs in real time during a remote telepresence session, and further comprising, receiving video information transmitted from the robot.

16. One or more computer-readable media having computer-executable instructions, which when executed perform steps, comprising, receiving information at a robot corresponding to skeletal data captured by an image and depth camera, and processing the information to control the operation of one or more components of the robotic device.

17. The one or more computer-readable media of claim 16 having further computer-executable instructions comprising, receiving information at the robot corresponding to facial expression data captured by an image and depth camera, and processing the information to control the operation of an expression system of the robotic device.

18. The one or more computer-readable media of claim 16 wherein the information comprises one or more action commands or data, or both action commands and data, computed at a remote device based upon the skeletal data.

19. The one or more computer-readable media of claim 16 wherein processing the information to control the operation of one or more components of the robotic device comprises controlling robot head movements, controlling robot manipulator movements, and controlling a robot mobility drive.

20. The one or more computer-readable media of claim 19 having further computer-executable instructions comprising, receiving information at the robot corresponding to facial expression data captured by an image and depth camera, and processing the information to control the operation of an expression system of the robotic device.