Output of a neural network method for deep odometry assisted by static scene optical flow

• US 10,552,979 B2
• Filed: 09/13/2017
• Issued: 02/04/2020
• Est. Priority Date: 09/13/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Zhu, Wentao, Wang, Yi, Luo, Yi
• Primary Examiner(s): Bernardi, Brenda C
• Application Number:
  US15/703,885
• Publication Number:
  US 20190080470A1
• Time in Patent Office: 874 Days
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/121 using special illumination

Abstract

A method of visual odometry for a non-transitory computer readable storage medium storing one or more programs is disclosed. The one or more programs includes instructions, which when executed by a computing device, causes the computing device to perform the following steps comprising: performing data alignment among sensors including a LiDAR, cameras and an IMU-GPS module; collecting image data and generating point clouds; processing, in the IMU-GPS module, a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds; and establishing an optical flow for visual odometry.

Claims

1. A method of visual odometry for a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, causes the computing device to perform the following steps comprising:

performing data alignment among sensors including a light detection and ranging (LiDAR) sensor, cameras, and an IMU-GPS module;

collecting image data and generating point clouds;

processing a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds;

establishing an optical flow for visual odometry;

receiving a first image of a first pair of image frames, and extracting representative features from the first image of the first pair in a first convolution neural network (CNN);

receiving a second image of the first pair, and extracting representative features from the second image of the first pair in the first CNN;

merging, in a first merge module, outputs from the first CNN;

decreasing feature map size in a second CNN;

generating a first flow output for each layer in a first deconvolution neural network (DNN); and

merging, in a second merge module, outputs from the second CNN and the first DNN to generate a first motion estimate.

2. The method according to claim 1 further comprising:

generating a second flow output for each layer in a second DNN, the second flow output serves as a first optical flow prediction.

3. The method according to claim 2 further comprising:

in response to the first motion estimate, generating a first set of motion parameters associated with the first pair in a recurrent neural network (RNN).

4. The method according to claim 3 further comprising:

training the visual odometry model by using at least one of the first optical flow prediction and the first set of motion parameters.

5. A method of visual odometry for a non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing device, causes the computing device to perform the following steps comprising:

performing data alignment among sensors including a light detection and ranging (LiDAR) sensor, cameras and an IMU-GPS module;

collecting image data and generating point clouds;

processing a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds;

establishing an optical flow for visual odometry;

receiving a first image of a second pair of image frames, and extracting representative features from the first image of the second pair in a first convolutional neural network (CNN);

receiving a second image of the second pair, and extracting representative features from the second image of the second pair in the first CNN;

merging, in a first merge module, outputs from the first CNN;

decreasing feature map size in a second CNN; and

generating a first flow output for each layer in a first deconvolutional neural network (DNN); and

merging, in the second merge module, outputs from the second CNN and the first DNN to generate a second motion estimate.

6. The method according to claim 5 further comprising:

generating a second flow output for each layer in the second DNN, the second flow output serves as a second optical flow prediction.

7. The method according to claim 6 further comprising:

in response to the second motion estimate and the first set of motion parameters, generating a second set of motion parameters associated with the second pair in the RNN.

8. The method according to claim 7 further comprising:

training the visual odometry model by using at least one of the second optical flow prediction and the second set of motion parameters.

9. A system for visual odometry, the system comprising:

an interne server, comprising;

an I/O port, configured to transmit and receive electrical signals to and from a client device;

a memory;

one or more processing units; and

one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions for;

performing data alignment among sensors including a light detection and ranging (LiDAR) sensor, cameras and an IMU-GPS module;

collecting image data and generating point clouds;

processing, in the IMU-GPS module, a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds;

establishing an optical flow for visual odometry;

receiving a first image of a first pair of image frames, and extracting representative features from the first image of the first pair in a first convolution neural network (CNN);

receiving a second image of the first pair and extracting representative features from the second image of the first pair in the first CNN;

merging, in a first merge module, outputs from the first CNN;

decreasing a feature map size in a second CNN;

generating a first flow output for each layer in a first deconvolution neural network (DNN); and

merging, in a second merge module, outputs from the second CNN and the first DNN to generate a first motion estimate.

10. The system according to claim 9 further comprising:

generating a second flow output for each layer in a second DNN, the second flow output serves as a first optical flow prediction.

11. The system according to claim 10 further comprising:

in response to the first motion estimate, generating a first set of motion parameters associated with the first pair in a recurrent neural network (RNN).

12. The system according to claim 11 further comprising:

training the visual odometry model by using at least one of the first optical flow prediction and the first set of motion parameters.

13. A system for visual odometry, the system comprising:

an interne server, comprising;

an I/O port, configured to transmit and receive electrical signals to and from a client device;

a memory;

one or more processing units; and

one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions for;

performing data alignment among sensors including a light detection and ranging (LiDAR) sensor, cameras and an IMU-GPS module;

collecting image data and generating point clouds;

processing, in the IMU-GPS module, a pair of consecutive images in the image data to recognize pixels corresponding to a same point in the point clouds;

establishing an optical flow for visual odometry;

receiving a first image of a second pair of image frames, and extracting representative features from the first image of the second pair in a first convolution neural network (CNN); and

receiving a second image of the second pair and extracting representative features from the second image of the second pair in the first CNN;

merging, in a first merge module, outputs from the first CNN;

decreasing feature map size in a second CNN;

generating a first flow output for each layer in a first deconvolutional neural network (DNN); and

merging, in the second merge module, outputs from the second CNN and the first DNN to generate a second motion estimate.

14. The system according to claim 13 further comprising:

generating a second flow output for each layer in the second DNN, the second flow output serves as a second optical flow prediction.

15. The system according to claim 14 further comprising:

in response to the second motion estimate and the first set of motion parameters, generating a second set of motion parameters associated with the second pair in the RNN.

16. The system according to claim 15 further comprising:

training the visual odometry model by using at least one of the second optical flow prediction and the second set of motion parameters.