Delayed Telop Aid

• US 20140324249A1
• Filed: 03/19/2014
• Published: 10/30/2014
• Est. Priority Date: 03/19/2013
• Status: Active Grant

• Current Assignee: Robotic Research OPCO LLC
• Original Assignee: Karl Murphy
• Inventors: Lacaze, Alberto Daniel, Murphy, Karl, Kluge, Karl
• Granted Patent: US 9,519,286 B2
• Time in Patent Office: Days
• US Class Current: 701/2
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246

Abstract

The proposed system, Delayed Telop Aid (DTA), improves the teleoperator'"'"'s ability to control the vehicle in a three step process. First, DTA predicts robot motion given the operators commands. Second, DTA creates synthetic images to produce a video feed that looks as if the robot communication link had no delay and no reduced bandwidth. Finally, DTA performs closed loop control on the robot platform to ensure that the robot follows the operator'"'"'s commands. A closed loop control of the platform makes sure that the predicted pose after the delay (and therefore the image presented to the operator) is achieved by the platform. This abstracts away the latency-sensitive parts of the robot control, making the robot'"'"'s behavior stable in the presence of poorly characterized latency between the operator and the vehicle.

Claims

1. A method for the teleoperation of robots, stored and executed by a computer platform, comprising the steps of:

predicting robot motion given the operators commands;

creating synthetic images to produce a video feed that looks as if the robot communication link had no delay and no reduced bandwidth; and

performing closed loop control on the robot platform to ensure that the robot follows the predicted robot motion.

2. The method of claim 1, wherein

the closed loop control of the platform makes sure that the predicted pose after the delay, and therefore the image presented to the operator, is achieved by the platform.

3. The method of claim 1, wherein

the work to perform these tasks is split between two sets of computing hardware, one at the platform and the second at a Operator Control Unit (OCU).

4. The method of claim 1, further comprising the steps of

extracting sufficient information about the structure of the scene using image flow and other sensor specific techniques to reconstruct a useful predicted view of the world at a future point in time;

using this reconstruction as a teleoperation aid;

making it appear that the communication delay between the OCU and the robotic platform does not exist;

creating the illusion that the operator is directly controlling the vehicle without the significant video and control delays; and

the feedback the operator receives from the video display would respond immediately to his actions.

5. The method of claim 1, further comprising the steps of

creating a 3D model of the environment;

and

accurately controlling the vehicle to the predicted pose.

6. The method of claim 5, further comprising the steps of

providing an OpenGL based image reconstruction algorithm;

providing a command interface to the robot platform based on distance and yaw commands; and

dividing the required image processing between the vehicle and the OCU, taking into consideration the video quality and computational assets available.

7. The method of claim 6, wherein the algorithm is chosen from one of the following for the group comprising:

SIFT;

SURF feature detection and tracking;

Lucas-Kanade method;

combined SURF feature tracking with Delaunay mesh generation;

Kalman filtering techniques; and

thin plate spline interpolation and curve fitting for synthetic image reconstruction.

8. The method of claim 6, wherein one or more image reconstruction techniques are chosen from the group comprising:

Delaunay mesh generation, thin plate spline interpolation, and curve fitting.

9. The method of claim 4, further comprising the steps of

generating predicted synthetic images;

once the features were identified and tracked from one frame to the next;

using these tracked features to create future synthetic video frames; and

generating a simple OpenGL based simulated display integrated with the OCU and a real platform.

10. The method of claim 1, further comprising the steps of

collecting raw Bayer images (1260×

980) with GMT timestamps at 5 Hz

collecting the (x, y, z) location and (r, p, y) orientation of the platform along with GMT timestamps at 50 Hz;

applying the appropriate coordinate transform to the platform location to obtain the camera location and orientation.

11. The method of claim 7, further comprising the steps of:

extracting SURF features for every frame;

projecting SURF features from the image into the world frame using navigation data and our fish-eye camera model;

performing ray intersection checks in 3D for the ray originating at the camera location and going through the feature location in world frame;

comparing each feature in the last frame is with every feature in the current frame;

if the rays are close enough, a 3D location for that feature is computed. (Condition comparing the similarity of the SURF descriptors of the two features in question to increase the tracking accuracy; and

if both conditions are satisfied, the track is considered good and added as a node to be used during the image reconstruction phase.

12. The method of claim 11, further comprising the steps of:

checking for mismatched SURF descriptors; and

only allowing pixels with reasonable shifts in the image to pass through.

13. The method of claim 12, further comprising the steps of:

using a Kalman filter for each feature being tracked to improve the estimate of its 3D location using its past history;

the Kalman filter is initialized with a best estimate of the 3D location of the feature;

each iteration the filter is updated using the (x, y) location of the tracked pixel in the current image;

internally, the filter keeps track of the 3D state (x, y, z) of the feature and updates it based on recorded measurements;

at each iteration the filter is updated using the (x, y) location of the tracked pixel in the current image;

internally, the filter keeps track of the 3D state (x, y, z) of the feature and updates it based on recorded measurements;

the 3D locations of the tracked features can be sent to the OCU along with the compressed image and the associated navigation state information of the platform;

creating a synthetic image using one or more of the image reconstruction techniques.

14. The method of claim 13, further comprising the steps of:

interpolating surrounding flow vectors;

generating a triangular Delaunay mesh connecting the image features with known optical flow vectors;

the regions between are mapped onto triangular regions;

this surface, with the decompressed image from the platform texture mapped to it, will then be deformed to produce the reconstructed image for the operator to drive by; and

the predicted future locations of the tracked features are used to deform the Delaunay mesh.

15. The method of claim 14, further comprising the steps of:

as the operator steers, additional commands go into the history queue and to the vehicle itself;

to simulate these joystick commands when working with recorded data, future navigation data is read from collected datasets;

this produces a new future vehicle position estimate, and a new view position estimate for each feature in the current image;

the image is treated as a rubber sheet with the locations of the tracked image features as the control points;

when the image features are repositioned in the field of view based on the vehicle motion prediction, the portions of the image not directly associated with image features are interpolated based on the new positions of the points in the triangular mesh;

the compressed image is texture mapped onto this warped surface;

these images are then displayed on the screen.

16. The method of claim 11, further comprising the steps of:

a point on the mesh close to the source feature will be moved to a place close to the corresponding target feature location;

the points in between are interpolated smoothly using Bookstein'"'"'s thinplate spline algorithm;

generating two lists consisting of the pixel locations of the tracked features in the current image and the predicted future 2D pixel locations of the tracked features;

the predicted future pixel locations of the tracked features are computed by projecting the 3D feature location back into the synthetic 2D image using the world-to-image transformation provided by the camera model using the predicted navigation state of the platform;

the coefficients of an interpolated cubic function are computed;

once the parameters are estimated, new (x, y) locations of all the pixels in the synthetic future image are computed;

this image is then texture mapped to a surface in OpenGL and displayed to the operator.

17. The method of claim 16, wherein the bending of the thin plate can be controlled using a Rigidity parameter.

18. The method of claim 1, further comprising the steps of:

sending the robot desired odometer and yaw commands from the joystick;

taking the last video frame received from the platform and texture mapping it on a flat surface;

using the predicted pose of the platform, computed using the commands sent to the platform from the OCU);

using orthographic projection;

displaying the synthetic image to the operator.

19. The method of claim 18, further comprising the steps of:

the odometer and yaw commands sent to the platform using the joystick are queued up in the OCU along with a timestamp;

the image reconstruction module has direct access to this queue of commands and uses the commands right away to provide instant feedback to the operator by creating predicted synthetic frames based on those commands and the last video frame/vehicle status received from the platform;

the video frames and vehicle status messages received from the platform are also queued up in the OCU;

in order to simulate latency, the camera frames are sent to the image reconstruction module after a predefined time interval has passed (from the time they are taken);

this delay threshold is configurable to simulate different latencies;

when the image reconstruction module receives a new frame, the corresponding vehicle status message is found by matching the frame id stored in the vehicle status message; and

this correction in pose is used when the inventors generate the next synthetic image for the operator.

20. The method of claim 19, further comprising:

a Sensor Data Capture and Processing Module running on the platform;

a Vehicle Command Module provides an interface to send commands from joysticks or other devices to the Image Reconstruction Module as well as to the Vehicle Control Module;

a Platform Pose Prediction sub module evaluates the commands sent to the platform on model of the platform and the world around the platform;

a Vehicle Control Module resides on the platform and can be used to talk to the platform'"'"'s drive by wire interface;

a Navigation Data Interface provides system navigation state information to other modules running on the platform; and

a World Modeling Module will receive data from the Sensor Data Capture and Processing Module as well as the Navigation Data Interface.