Delayed telop aid

• US 9,519,286 B2
• Filed: 03/19/2014
• Issued: 12/13/2016
• Est. Priority Date: 03/19/2013
• Status: Active Grant

• Current Assignee: Robotic Research OPCO LLC
• Original Assignee: Robotic Research LLC
• Inventors: Lacaze, Alberto Daniel, Murphy, Karl, Kluge, Karl
• Primary Examiner(s): Butler, Rodney
• Assistant Examiner(s): BRUSHABER, FREDERICK M
• Application Number:
  US14/219,834
• Publication Number:
  US 20140324249A1
• Time in Patent Office: 1,000 Days
• Field of Search: 701/2
• US Class Current: 1/1
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246

Abstract

The proposed system, Delayed Telop Aid (DTA), improves the teleoperator'"'"'s ability to control the vehicle in a three step process. First, DTA predicts robot motion given the operators commands. Second, DTA creates synthetic images to produce a video feed that looks as if the robot communication link had no delay and no reduced bandwidth. Finally, DTA performs closed loop control on the robot platform to ensure that the robot follows the operator'"'"'s commands. A closed loop control of the platform makes sure that the predicted pose after the delay (and therefore the image presented to the operator) is achieved by the platform. This abstracts away the latency-sensitive parts of the robot control, making the robot'"'"'s behavior stable in the presence of poorly characterized latency between the operator and the vehicle.

Claims

1. A method for the teleoperation of robots, stored and executed by a computer platform, comprising the steps of:

predicting robot motion from an operator'"'"'s queued commands to an Operator Control Unit (OCU);

creating synthetic images to produce a video feed that looks as if the robot communication link had no delay and no reduced bandwidth;

controlling the robot'"'"'s movement using closed loop; and

displaying the synthetic image to an operator

creating a 3D model of the environment;

accurately controlling the vehicle to the predicted pose;

providing an OpenGL based image reconstruction algorithm;

providing a command interface to the robot for receiving distance and yaw input commands;

dividing the required image processing between the vehicle and the OCU, taking into consideration the video quality and computational assets available;

wherein the algorithm is chosen from one of the following for the group comprising;

SIFT;

a SURF feature for detection and tracking;

Lucas-Kanade method;

combined SURF feature tracking with Delaunay mesh generation;

Kalman filtering techniques; and

thin plate spline interpolation and curve fitting for synthetic image reconstruction;

extracting SURF features for every frame;

projecting SURF features from the image into the world frame using navigation data and our fish-eye camera model;

performing ray intersection checks in 3D to determine if the ray originating at the camera location goes through the feature location in world frame;

comparing each feature in the last frame is with every feature in the current frame;

if the rays are close enough, a 3D location for that feature is computed;

comparing the similarity of the SURF descriptors of the features in question, to each other, to increase the tracking accuracy; and

if both conditions are satisfied, the track is considered good and added as a node to be used during an image reconstruction phase.

2. The method of claim 1, wherein

the closed loop control of the robot makes sure that the predicted pose after the delay, and therefore the image presented to the operator, is achieved by the robot.

3. The method of claim 1, wherein

the work to perform these tasks is split between two sets of computing hardware, one at the robot and the second at the OCU.

4. The method of claim 1, further comprising the steps of

extracting sufficient information about the structure of the scene using image flow and other sensor specific techniques to reconstruct a useful predicted view of the world at a future point in time;

using this reconstruction as a teleoperation aid;

making it appear that the communication delay between the OCU and the robot does not exist;

creating the illusion that the operator is directly controlling the vehicle without the significant video and control delays; and

the feedback the operator receives from the video display would respond immediately to his actions.

5. The method of claim 4, further comprising the steps of

generating predicted synthetic images;

once the features were identified and tracked from one frame to the next;

using these tracked features to create future synthetic video frames; and

generating a simple OpenGL based simulated display integrated with the OCU and a real robot.

6. The method of claim 1, wherein one or more image reconstruction techniques are chosen from the group comprising:

Delaunay mesh generation, thin plate spline interpolation, and curve fitting.

7. The method of claim 1, further comprising the steps of

collecting raw Bayer images (1260×

980) with GMT timestamps at 5 Hz;

collecting the (x, y, z) location and (r, p, y) orientation of the robot along with GMT timestamps at 50 Hz; and

applying the appropriate coordinate transform to the robot location to obtain the camera location and orientation.

8. The method of claim 1, further comprising the steps of:

checking for mismatched SURF descriptors; and

only allowing pixels with shifts in the image to pass through.

9. The method of claim 8, further comprising the steps of:

using a Kalman filter for each feature being tracked to improve the estimate of its 3D location using its past history;

the Kalman filter is initialized with a best estimate of the 3D location of the feature;

each iteration the filter is updated using the (x, y) location of the tracked pixel in the current image;

internally, the filter keeps track of the 3D state (x, y, z) of the feature and updates it based on recorded measurements;

at each iteration the filter is updated using the (x, y) location of the tracked pixel in the current image;

internally, the filter keeps track of the 3D state (x, y, z) of the feature and updates it based on recorded measurements;

the 3D locations of the tracked features can be sent to the OCU along with the compressed image and the associated navigation state information of the robot;

creating a synthetic image using one or more of the image reconstruction techniques.

10. The method of claim 9, further comprising the steps of:

interpolating surrounding flow vectors;

generating a triangular Delaunay mesh connecting the image features with known optical flow vectors;

the regions between are mapped onto triangular regions;

this surface, with the decompressed image from the robot texture mapped to it, will then be deformed to produce the reconstructed image for the operator to drive by; and

the predicted future locations of the tracked features are used to deform the Delaunay mesh.

11. The method of claim 10, further comprising the steps of:

as the operator steers, additional commands go into a history queue and to the vehicle itself;

to simulate these commands when working with recorded data, future navigation data is read from collected datasets;

this produces a new future vehicle position estimate, and a new view position estimate for each feature in the current image;

the image is treated as a rubber sheet with the locations of the tracked image features as the control points;

when the image features are repositioned in the field of view based on the vehicle motion prediction, the portions of the image not directly associated with image features are interpolated based on the new positions of the points in the triangular mesh;

the compressed image is texture mapped onto this warped surface;

these images are then displayed on the screen.

12. The method of claim 1, further comprising the steps of:

a point on the mesh close to the source feature will be moved to a place close to the corresponding target feature location;

the points in between are interpolated smoothly using Bookstein'"'"'s thinplate spline algorithm;

generating two lists consisting of the pixel locations of the tracked features in the current image and the predicted future 2D pixel locations of the tracked features;

the predicted future pixel locations of the tracked features are computed by projecting the 3D feature location back into the synthetic 2D image using the world-to-image transformation provided by the camera model using the predicted navigation state of the robot;

the coefficients of an interpolated cubic function are computed;

once the parameters are estimated, new (x, y) locations of all the pixels in the synthetic future image are computed;

this image is then texture mapped to a surface in OpenGL and displayed to the operator.

13. The method of claim 12, wherein the bending of a thin plate can be controlled using a Rigidity parameter.

14. A method for the teleoperation of robots, stored and executed by a computer platform, comprising the steps of:

predicting robot motion given a operator'"'"'s commands;

creating synthetic images to produce a video feed that looks as if the robot communication link had no delay and no reduced bandwidth;

controlling the robot'"'"'s movement using closed loop;

sending the robot desired odometer and yaw commands from a joystick;

taking the last video frame received from the robot and texture mapping it on a flat surface;

using the predicted pose of the robot, computed using the commands sent to the robot from an Operator Control Unit (OCU);

using orthographic projection;

displaying the synthetic image to the operator;

the odometer and yaw commands sent to the robot using the joystick are queued up in the OCU along with a timestamp;

an image reconstruction module has direct access to this queue of commands and uses the commands right away to provide instant feedback to the operator by creating predicted synthetic frames based on those commands and the last video frame/vehicle status received from the robot;

the video frames and vehicle status messages received from the robot are also queued up in the OCU;

in order to simulate latency, the camera frames are sent to the image reconstruction module after a predefined time interval has passed from the time they are taken;

this delay threshold is configurable to simulate different latencies;

when the image reconstruction module receives a new frame, the corresponding vehicle status message is found by matching the frame id stored in the vehicle status message; and

this correction in pose is used when the inventors generate the next synthetic image for the operator.

15. The method of claim 14, further comprising:

a Sensor Data Capture and Processing Module running on the robot;

a Vehicle Command Module provides an interface to send commands from joysticks or other devices to the Image Reconstruction Module as well as to the Vehicle Control Module;

a robot Pose Prediction sub module evaluates the commands sent to the robot on model of the robot and the world around the robot;

a Vehicle Control Module resides on the robot and can be used to talk to the robot'"'"'s drive by wire interface;

a Navigation Data Interface provides system navigation state information to other modules running on the robot; and

a World Modeling Module will receive data from the Sensor Data Capture and Processing Module as well as the Navigation Data Interface.