Neural network architecture system for deep odometry assisted by static scene optical flow

• US 10,671,083 B2
• Filed: 09/13/2017
• Issued: 06/02/2020
• Est. Priority Date: 09/13/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Inventors: Zhu, Wentao, Wang, Yi, Luo, Yi
• Primary Examiner(s): Nguyen, Kira
• Application Number:
  US15/703,879
• Publication Number:
  US 20190079534A1
• Time in Patent Office: 993 Days
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G01C 22/00 Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/025 Extracting rules from data
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 : using gradient-based methods
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks

Abstract

A system for visual odometry is disclosed. The system includes: an internet server, comprising: an I/O port, configured to transmit and receive electrical signals to and from a client device; a memory; one or more processing units; and one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions for: extracting representative features from a pair input images in a first convolution neural network (CNN) in a visual odometry model; merging, in a first merge module, outputs from the first CNN; decreasing feature map size in a second CNN; generating a first flow output for each layer in a first deconvolution neural network (DNN); merging, in a second merge module, outputs from the second CNN and the first DNN; generating a second flow output for each layer in a second DNN; and reducing accumulated errors in a recurrent neural network (RNN).

Claims

1. A system for visual odometry, the system comprising:

an internet server, comprising;

an I/O port, configured to transmit and receive electrical signals to and from a client device;

a memory;

one or more processing units; and

one or more programs stored in the memory and configured for execution by the one or more processing units, the one or more programs including instructions for;

extracting representative features from a pair of input images in a first convolution neural network (CNN) in a visual odometry model;

merging, in a first merge module, outputs from the first CNN;

decreasing a feature map size in a second CNN;

generating a first flow output for each layer in a first deconvolution neural network (DNN);

merging, in a second merge module, outputs from the second CNN and the first DNN;

generating, by the second merge module from the first flow of the first DNN and outputs from the second CNN, a motion estimate between the pair of input images;

generating a second flow output for each layer in a second DNN; and

reducing accumulated errors in a recurrent neural network (RNN).

2. The system according to claim 1, wherein the outputs from the first CNN include the representative features of a first image of the pair and the representative features of a second image of the pair.

3. The system according to claim 1, wherein the merging the outputs from the first CNN comprises:

using one of a patch-wise correlation or a simple concatenation during merging.

4. The system according to claim 1 further comprising:

generating a set of motion parameters in the RNN in response to the motion estimate and another set of motion parameters associated with an immediately previous pair of input images.

5. The system according to claim 4 further comprising:

providing the set of motion parameters associated with the pair of input images to the RNN.

6. The system according to claim 4 further comprising:

training the visual odometry model by using the set of motion parameters.

7. The system according to claim 1, wherein the first flow output has a first resolution, and the second flow output has a second resolution higher than the first resolution.

8. The system according to claim 7 further comprising:

training the visual odometry model by using the second flow output; and

generating the motion parameters based on the motion estimate.

9. A method of visual odometry, comprising:

extracting representative features from a pair of input images in a first convolution neural network (CNN) in a visual odometry model;

merging, in a first merge module, outputs from the first CNN;

decreasing a feature map size in a second CNN;

generating a first flow output for each layer in a first deconvolution neural network (DNN);

merging, in a second merge module, outputs from the second CNN and the first DNN;

generating, by the second merge module from the first flow of the first DNN and outputs from the second CNN, a motion estimate between the pair of input images;

generating a second flow output for each layer in a second DNN; and

reducing accumulated errors in a recurrent neural network (RNN).

10. The method according to claim 9, wherein the outputs from the first CNN include the representative features of a first image of the pair and the representative features of a second image of the pair.

11. The method according to claim 9, wherein the merging the outputs from the first CNN comprises:

using one of a patch-wise correlation or a simple concatenation during merging.

12. The method according to claim 9, further comprising:

generating a set of motion parameters in the RNN in response to the motion estimate and another set of motion parameters associated with an immediately previous pair of input images.

13. The method according to claim 12, further comprising:

providing the set of motion parameters associated with the pair of input images to the RNN.

14. The method according to claim 12, further comprising:

training the visual odometry model by using the set of motion parameters.

15. The method according to claim 9, wherein the first flow output has a first resolution, and the second flow output has a second resolution higher than the first resolution.

16. The method according to claim 15, further comprising:

training the visual odometry model by using the second flow output; and

generating the motion parameters based on the motion estimate.