Robotic camera control via motion capture

• US 10,427,305 B2
• Filed: 07/21/2016
• Issued: 10/01/2019
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Autodesk, Inc.
• Original Assignee: Autodesk, Inc.
• Inventors: Atherton, Evan Patrick, Thomasson, David, Conti, Maurice Ugo, Kerrick, Heather
• Primary Examiner(s): Tran, Khoi H
• Assistant Examiner(s): Peche, Jorge O
• Application Number:
  US15/216,583
• Publication Number:
  US 20180021956A1
• Time in Patent Office: 1,167 Days
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 9/16 Programme controls programme controls in general G05B19/00, e.g. numerical pogramme controls G05B19/18; recording or playback systems G05B19/42
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1679 characterised by the tasks executed
• CPC Class Codes: B25J 9/1689 Teleoperation
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/35444 Gesture interface, controlled machine observes operator, executes commands
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A motion capture setup records the movements of an operator, and a control engine then translates those movements into control signals for controlling a robot. The control engine may directly translate the operator movements into analogous movements to be performed by the robot, or the control engine may compute robot dynamics that cause a portion of the robot to mimic a corresponding portion of the operator.

Claims

1. A computer-implemented method for controlling a robot to generate multi-media content, the method comprising:

generating, via a first set of one or more sensors, motion capture data based on one or more movements of an operator;

processing the motion capture data to generate control signals for controlling the robot, wherein a second set of one or more sensors is coupled to the robot;

transmitting the control signals to the robot to cause the robot to mimic the one or more movements of the operator; and

generating content associated with the one or more movements of the operator based on a plurality of image frames of the robot that are captured by the second set of one or more sensors.

2. The computer-implemented method of claim 1, wherein generating the motion capture data comprises tracking at least one of head, body, and limb movements associated with the operator via the first set of one more sensors.

3. The computer-implemented method of claim 1, wherein processing the motion capture data to generate the control signals comprises:

modeling a first set of dynamics of the operator that corresponds to the one or more movements;

determining a second set of dynamics of the robot that corresponds to the one or more movements; and

generating the control signals based on the second set of dynamics.

4. The computer-implemented method of claim 1, wherein causing the robot to mimic the one or more movements of the operator comprises actuating a set of joints included in the robot to perform an articulation sequence associated with the one or more movements.

5. The computer-implemented method of claim 4, wherein the robot comprises a robotic arm that includes the set of joints.

6. The computer-implemented method of claim 1, wherein causing the robot to mimic the one or more movements comprises causing only a first portion of the robot to mimic only a subset of the one or more movements.

7. The computer-implemented method of claim 1, further comprising capturing sensor data while the robot mimics the one or more movements.

8. The computer-implemented method of claim 1, wherein the one or more movements comprise a target trajectory for the second set of one or more sensors coupled to the robot.

9. One or more non-transitory computer-readable media that, when executed by one or more processors, causes the one or more processors to control a robot in order to generate multi-media content, by performing the steps of:

generating, via a first set of one or more sensors, motion capture data based on one or more movements of an operator;

processing the motion capture data to generate control signals for controlling the robot, wherein a second set of one or more sensors is coupled to the robot;

transmitting the control signals to the robot to cause the robot to mimic the one or more movements of the operator; and

generating content associated with the one or more movements of the operator based on a plurality of image frames of the robot that are captured by the second set of one or more sensors.

10. The one or more non-transitory computer-readable media of claim 9, wherein generating the motion capture data comprises tracking at least one of head, body, and limb movements associated with the operator via the first set of one more sensors.

11. The one or more non-transitory computer-readable media of claim 9, wherein processing the motion capture data to generate the control signals comprises:

modeling a first set of dynamics of the operator that corresponds to the one or more movements;

determining a second set of dynamics of the robot that corresponds to the one or more movements; and

generating the control signals based on the second set of dynamics.

12. The one or more non-transitory computer-readable media of claim 9, wherein causing the robot to mimic the one or more movements of the operator comprises actuating a set of joints included in the robot to perform an articulation sequence associated with the one or more movements.

13. The one or more non-transitory computer-readable media of claim 9, wherein causing the robot to mimic the one or more movements comprises causing only a first portion of the robot to mimic only a subset of the one or more movements.

14. The one or more non-transitory computer-readable media of claim 9, further comprising capturing sensor data while the robot mimics the one or more movements.

15. The one or more non-transitory computer-readable media of claim 9, wherein the one or more movements comprise a target trajectory for the second set of one or more sensors coupled to the robot.

16. A system for controlling a robot to generate multi-media content, comprising:

a memory storing a control application; and

a processor coupled to the memory and configured to;

generate, via a first set of one or more sensors, motion capture data based on one or more movements of an operator,

process the motion capture data to generate control signals for controlling the robot, wherein a second set of one or more sensors is coupled to the robot,

transmit the control signals to the robot to cause the robot to mimic the one or more movements of the operator, and

generate content associated with the one or more movements of the operator based on a plurality of image frames of the robot that are captured by the second set of one or more sensors.

17. The system of claim 16, wherein the processor is configured to execute the control application to:

generate the motion capture data;

process the motion capture data to generate the control signals; and

transmit the control signals to the robot.