System and method for real-time large image homography processing

• US 10,497,145 B2
• Filed: 11/16/2017
• Issued: 12/03/2019
• Est. Priority Date: 11/16/2017
• Status: Active Grant

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: Alesiani, Francesco, Singh, Ankur
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/814,437
• Publication Number:
  US 20190147621A1
• Time in Patent Office: 747 Days
• Field of Search: None
• CPC Class Codes: G05B 17/02 electric
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method for image processing performed by a computing device includes: (a) receiving one or more images and preprocessing the one or more images by extracting one or more features from the one or more images; (b) estimating image homographies (and/or fundamental matrices) based on the one or more features extracted from the one or more images; (c) determining particle trajectories based on composition of the image homographies (and/or fundamental matrices), wherein a particle trajectory comprises image pixel points identified in a subset of the one or more images; and (d) determining a joint estimation of camera pose, camera parameters, and a dense depth map using the particle trajectories.

Claims

1. A method for image processing performed by a computing device, the method comprising:

receiving one or more images and preprocessing the one or more images by extracting one or more features from the one or more images;

estimating image homographies and/or fundamental matrices based on the one or more features extracted from the one or more images;

determining particle trajectories based on composition of the image homographies and/or fundamental matrices, wherein a particle trajectory comprises image pixel points identified in a subset of the one or more images; and

determining a joint estimation of camera pose, camera parameters, and a dense depth map using the particle trajectories.

2. The method according to claim 1, further comprising:

clustering the particle trajectory for object tracking and object detection.

3. The method according to claim 1, further comprising:

propagating an object classification movement hypothesis based on the image homographies and/or fundamental matrices.

4. The method according to claim 1, wherein in the image homographies and/or fundamental matrices, each image homography and/or fundamental matrix comprises a hypothesis flag and/or a selection mask, the hypothesis flag indicating a consistency of the homography and the selection mask indicating one or more parts of the one or more images for which the image homography applies.

5. The method according to claim 4, wherein the hypothesis flag is implemented with a softmax output.

6. The method according to claim 1, wherein the one or more images are preprocessed with a convolutional neural network.

7. The method according to claim 6, wherein the convolutional neural network is a ResNet, a ResNext, an AlexNet, or a VGG16.

8. The method according to claim 6, wherein each image is preprocessed only once.

9. The method according to claim 1, wherein estimating the image homographies and/or fundamental matrices based on the one or more features extracted from the one or more images comprises reconstructing an extracted feature at different scale levels using a feature pyramid network.

10. The method according to claim 1, wherein estimating each image homography and/or fundamental matrix comprises obtaining an image homography and/or a fundamental matrix based on a regression network, wherein a last layer of the regression network has a linear activation function.

11. The method according to claim 10, wherein a loss function for the regression network is a mean square error norm.

12. An image processor comprising:

a convolutional neural network configured to;

receive one or more images, and

preprocess the one or more images by extracting one or more features from the one or more images; and

a task-specific head convolutional neural network configured to;

estimate image homographies and/or fundamental matrices based on the one or more features extracted from the one or more images,

determine particle trajectories based on composition of the image homographies and/or fundamental matrices, wherein a particle trajectory comprises image pixel points identified in a subset of the one or more images, and

determine a joint estimation of camera pose, camera parameters, and a dense depth map using the particle trajectories.

13. The image processor according to claim 12, wherein the task-specific head is further configured to:

cluster the particle trajectories for object tracking and object detection.

14. The image processor according to claim 12, wherein the task-specific head is further configured to:

propagate an object classification movement hypothesis based on the image homographies and/or fundamental matrices.

15. The image processor according to claim 12, wherein each image homography and/or fundamental matrix comprises a hypothesis flag and/or a selection mask, the hypothesis flag indicating a consistency of the homography and/or fundamental matrix and the selection mask indicating one or more parts of the one or more images for which the image homography and/or fundamental matrix applies.

16. A non-transitory computer-readable medium having processor-executable instructions stored thereon, wherein the processor-executable instructions, when executed by an image processor, facilitate:

receiving one or more images and preprocessing the one or more images by extracting one or more features from the one or more images;

estimating image homographies and/or fundamental matrices based on the one or more features extracted from the one or more images;

determining particle trajectories based on composition of the image homographies and/or fundamental matrices, wherein a particle trajectory comprises image pixel points identified in a subset of the one or more images; and

determining a joint estimation of camera pose, camera parameters, and a dense depth map using the particle trajectories.

17. The non-transitory computer-readable according to claim 16, wherein the processor-executable instructions, when executed, further facilitate:

clustering the particle trajectory for object tracking and object detection.

18. The non-transitory computer-readable according to claim 16, wherein the processor-executable instructions, when executed, further facilitate:

propagating an object classification movement hypothesis based on the image homographies and/or fundamental matrices.

19. The non-transitory computer-readable according to claim 16, wherein in the image homographies and/or fundamental matrices, each image homography and/or fundamental matrix comprises a hypothesis flag and/or a selection mask, the hypothesis flag indicating a consistency of the homography and the selection mask indicating one or more parts of the one or more images for which the image homography applies.