Optical flow estimation using a neural network and egomotion optimization

• US 10,262,224 B1
• Filed: 07/19/2017
• Issued: 04/16/2019
• Est. Priority Date: 07/19/2017
• Status: Active Grant

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: The United States of America As Represented By The Secretary of Agriculture
• Inventors: Harguess, Joshua D., Barngrover, Christopher M., Rahimi, Amin M.
• Primary Examiner(s): Yang, Qian
• Application Number:
  US15/654,573
• Time in Patent Office: 636 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2414 Smoothing the distance, e.g. radial basis function networks [RBFN]
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position

Abstract

A camera is connected to a trained neural network. The camera takes an image of a scene and transmits the image to the neural network. A processor connected to the neural network has a localization filter and a robot model implemented therein. A global positioning system (GPS) receiver and inertial measurement unit (IMU) transmit GPS information and IMU information, respectively, to the processor. The localization filter filters the received GPS and IMU information and inputs the filtered information into the robot model. The robot model outputs current position information corresponding to the current image and previous position information corresponding to the respective one or more previous images. The neural network uses the current image and associated current position information and the one or more previous images and respective associated previous position information to generate an estimated optical flow image, which is transmitted to an object detection system.

Claims

1. A method comprising the steps of:

providing a processor having a neural network, localization filter, and robotic model implemented therein, wherein the processor is operatively connected to a camera connected to a platform, wherein one or more of the camera and the platform is moving;

using the camera to capture a current image of a scene at a current time, wherein the scene includes one or more moving objects therein;

transmitting the current image and one or more previous images of the scene to the neural network, wherein the one or more previous images of the scene are taken at respective one or more previous times prior to the current time;

receiving, at the neural network, current position information and previous position information from the robot model, wherein the current position information is the position of the camera at the current time and the previous position information is the one or more positions of the camera at the respective one or more previous times; and

generating, using the neural network, an estimated optical flow image using the current image of the scene, the current position information, one or more previous images of the scene, and the previous position information;

wherein, prior to the providing step, training the neural network using a ground truth optical flow image, one or more training images of a scene having one or more moving objects therein, and position information corresponding to positions of the camera when the one or more training images are taken; and

wherein the ground truth optical flow image includes a flow of salient objects but does not include a flow of other pixels even when the other pixels visually flow due to movement of the camera.

2. The method of claim 1 further comprising the step of transmitting the estimated optical flow image from the neural network to an object detection system operatively connected to the processor.

3. The method of claim 1, wherein the processor is operatively connected to a global positioning system (GPS) receiver and an inertial measurement unit (IMU), the method further comprising the step of transmitting, from the GPS receiver and the IMU, GPS information and IMU information, respectively, to the processor.

4. The method of claim 3 further comprising the step of, using the localization filter implemented within the processor to filter the received GPS information and IMU information and input the filtered GPS information and the filtered IMU information into the robot model.

5. The method of claim 4 further comprising the step of using the processor to output to the neural network, from the robot model, current position information corresponding to the current image and previous position information corresponding to the respective one or more previous images.

6. The method of claim 5 further comprising the step of generating, using the neural network, an estimated optical flow image using the current image and the associated current position information while ignoring a motion field due to ego-motion, and the one or more previous images and the respective associated previous position information.

7. The method of claim 1, wherein the step of generating, using the neural network, an estimated optical flow image is performed by using an algorithm to apply weights to the current image of the scene, the current position information, one or more previous images of the scene, and the previous position information, wherein the weights are based upon previous training of the neural network.

8. The method of claim 1, wherein the step of generating, using the neural network, an estimated optical flow image comprises using one or more of information, data, images, and algorithms stored within memory within the processor.

9. A system comprising:

a neural network;

a camera operatively connected to the neural network, the camera configured to take an image of a scene and transmit the image of the scene to the neural network; and

a processor operatively connected to the neural network, the processor having a localization filter and a robot model implemented therein, wherein the processor is configured to output from the robot model current position information corresponding to the current image and previous position information corresponding to the respective one or more previous images,

wherein the neural network is configured to use the current image and the associated current position information, and the one or more previous images and the respective associated previous position information to generate an estimated optical flow image;

wherein training the neural network using a ground truth optical flow image, one or more training images of a scene having one or more moving objects therein, and position information corresponding to positions of the camera when the one or more training images are taken; and

wherein the ground truth optical flow image includes a flow of salient objects but does not include a flow of other pixels even when the other pixels visually flow due to movement of the camera.

10. The system of claim 9 further comprising object detection system operatively connected to the processor, wherein the processor is configured to transmit the estimated optical flow image from the neural network to the object detection system.

11. The system of claim 9, wherein the processor is operatively connected to a global positioning system (GPS) receiver and an inertial measurement unit (IMU), wherein the GPS receiver and the IMU are configured to transmit GPS information and IMU information, respectively, to the processor.

12. The system of claim 11, wherein the processor is further configured to use the localization filter implemented within the processor to filter the received GPS information and IMU information and input the filtered GPS information and filtered IMU information into the robot model.

13. The system of claim 12, wherein the processor is further configured to output to the neural network, from the robot model, current position information corresponding to the current image and previous position information corresponding to the respective one or more previous images.

14. The system of claim 13, wherein the processor is configured to use the neural network to generate an estimated optical flow image using the current image and the associated current position information while ignoring a motion field due to ego-motion, and the one or more previous images and the respective associated previous position information.

15. The system of claim 14, wherein the neural network is configured to determine an estimated optical flow image by using an algorithm to apply weights to the current image of the scene, the current position information, one or more previous images of the scene, and the previous position information, wherein the weights are based upon previous training of the neural network.

16. The system of claim 9, wherein the neural network is trained using a ground truth optical flow image, one or more training images of a scene having one or more moving objects therein, and position information corresponding to positions of the camera when the one or more training images are taken.

17. The system of claim 9, wherein the processor further comprises memory therein, wherein one or more of information, data, images, and/or algorithms is stored within the memory, wherein the processor is configured to use the stored information, data, images, and algorithms to generate, using the neural network, the estimated optical flow image.

18. A system comprising:

a neural network;

a camera operatively connected to the neural network, the camera configured to take an image of a scene and transmit the image of the scene to the neural network;

a processor operatively connected to the neural network, the processor having a localization filter and a robot model implemented therein;

a global positioning system (GPS) receiver operatively connected to the processor; and

an inertial measurement unit (IMU) operatively connected to the processor,

wherein the GPS receiver and the IMU are configured to transmit GPS information and IMU information, respectively, to the processor,

wherein the processor is configured to use the localization filter to filter the received GPS information and IMU information and input the filtered information into the robot model,

wherein the processor is configured to output from the robot model current position information corresponding to the current image and previous position information corresponding to the respective one or more previous images,

wherein the neural network is configured to use the current image and the associated current position information while ignoring a motion field due to ego-motion, and the one or more previous images and the respective associated previous position information to generate an estimated optical flow image;

wherein training the neural network using a ground truth optical flow image, one or more training images of a scene having one or more moving objects therein, and position information corresponding to positions of the camera when the one or more training images are taken; and

wherein the ground truth optical flow image includes a flow of salient objects but does not include a flow of other pixels even when the other pixels visually flow due to movement of the camera.