Feature based high resolution motion estimation from low resolution images captured using an array source

• US 9,813,616 B2
• Filed: 11/05/2015
• Issued: 11/07/2017
• Est. Priority Date: 08/23/2012
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Fotonation Cayman Limited
• Inventors: Lelescu, Dan, Jain, Ankit K.
• Primary Examiner(s): SPINKS, ANTOINETTE T
• Application Number:
  US14/933,871
• Publication Number:
  US 20160165134A1
• Time in Patent Office: 733 Days
• Field of Search: 3482181
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06T 3/4069 by subpixel displacements
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Systems and methods in accordance with embodiments of the invention enable feature based high resolution motion estimation from low resolution images captured using an array camera. One embodiment includes performing feature detection with respect to a sequence of low resolution images to identify initial locations for a plurality of detected features in the sequence of low resolution images, where the at least one sequence of low resolution images is part of a set of sequences of low resolution images captured from different perspectives. The method also includes synthesizing high resolution image portions, where the synthesized high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images. The method further including performing feature detection within the high resolution image portions to identify high precision locations for the detected features, and estimating camera motion using the high precision locations for said plurality of detected features.

Claims

1. A method for performing feature based high resolution motion estimation from a plurality of low resolution images, comprising:

performing feature detection with respect to a first sequence of low resolution images captured by a first imager in an imager array using a processor configured by software to identify initial locations for a plurality of detected features in the first sequence of low resolution images, where the first sequence of low resolution images is part of a set of sequences of low resolution images captured from different perspectives by different imagers in the imager array, a first plurality of images includes one image from each sequence of low resolution images taken by different imagers from different perspectives at a first point in time, and a second plurality of images includes one images from each sequence of low resolution images taken by different imagers from different perspectives at a second point in time;

synthesizing a first set of high resolution image portions from the set of sequences of low resolution images captured from different perspectives using the processor configured by software to perform a super-resolution process using the first plurality of images and parallax information, where the synthesized high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images;

synthesizing a second set of high resolution image portions from the set of sequences of low resolution images captured from different perspectives using the processor configured by software to perform a super-resolution process using the second plurality of images and parallax information, where the synthesized high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images;

performing feature detection within the first and second sets of high resolution image portions to identify locations for said plurality of detected features to a higher precision than the initial locations identified in the low resolution images using the processor configured by software; and

estimating camera motion using the high precision locations for said plurality of detected features using the processor configured by software.

2. The method of claim 1, wherein the detected features are selected from the group consisting of:

edges, corners, and blobs.

3. The method of claim 1, wherein performing feature detection with respect to a sequence of low resolution images further comprises:

detecting the location of features in a first frame from the low resolution sequence of images; and

detecting the location of features in a second frame from the low resolution sequence of images.

4. The method of claim 3, wherein detecting the location of features in a second frame from the sequence of low resolution images further comprises searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images.

5. The method of claim 4, wherein searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images further comprises:

identifying an image patch surrounding the location of the given feature in the first frame in the sequence of low resolution images; and

searching the second frame in the sequence of low resolution images for a corresponding image patch using a matching criterion.

6. The method of claim 5, wherein the matching criterion involves minimizing an error distance metric.

7. The method of claim 3, wherein performing feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features further comprises searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images using the high resolution image regions containing the features from the first frame in the low resolution sequence of images.

8. The method of claim 7, wherein searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images further comprises comparing high resolution image regions containing features from the second frame in the sequence of low resolution images to the high resolution image portions containing the features from the first frame in the sequence of low resolution images using a matching criterion.

9. The method of claim 8, wherein the matching criterion involves minimizing an error distance metric.

10. The method of claim 1, wherein the processor is part of an array camera that further comprises an imager array, the method further comprising capturing at least a plurality of the sequences of low resolution images in the set of sequences of low resolution images from different perspectives using the imager array.

11. The method of claim 1, wherein the high precision locations for said plurality of detected features estimate feature location at a subpixel precision relative to the size of the pixels of the frames in the sequence of low resolution images.

12. An array camera configured to perform feature based high resolution motion estimation from low resolution images captured using the array camera, comprising:

an imager array comprising a plurality of imagers;

a processor configured by software to control various operating parameters of the imager array;

wherein the software further configures the processor to;

capture a set of sequences of low resolution images captured from different perspectives using the imager array, where a first imager captures a first sequence of low resolution images from a first perspective, a second imager captures a second sequence of low resolution images from a second perspective, a first plurality of images includes one image from each sequence of low resolution images taken by different imagers from different perspectives at a first point in time, and a second plurality of images includes one images from each sequence of low resolution images taken by different imagers from different perspectives at a second point in time;

perform feature detection with respect to one of the set of sequences of low resolution images to identify initial locations for a plurality of detected features in the sequence of low resolution images;

synthesize a first set of high resolution image portions from the set of sequences of low resolution images captured from different perspectives using a super-resolution process using the first plurality of images and parallax information, where the high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images;

synthesize a second set of high resolution image portions from the set of sequences of low resolution images captured from different perspectives using a super-resolution process using the second plurality of images and parallax information, where the synthesized high resolution image portions contain the identified plurality of detected features from the sequence of low resolution images;

perform feature detection within the first and second sets of high resolution image portions to identify locations for said plurality of detected features to a higher precision than the initial locations identified in the low resolution images; and

estimate camera motion using the high precision locations for said plurality of detected features.

13. The array camera of claim 12, where the detected features are selected from the group consisting of:

edges, corners, and blobs.

14. The array camera of claim 12, wherein the processor is further configured to perform feature detection with respect to a sequence of low resolution images by:

detecting the location of features in a first frame from the sequence of low resolution images; and

detecting the location of features in a second frame from the sequence of low resolution images.

15. The array camera of claim 14, wherein the processor is further configured by software to detect the location of features in a second frame from the sequence of low resolution images by searching the second frame from the sequence of low resolution images to locate features detected in the first frame from the sequence of low resolution images.

16. The array camera of claim 15, wherein the processor is further configured by software to search a second frame from the sequence of low resolution images to locate a given feature detected in the first frame from the sequence of low resolution images by:

identifying an image patch surrounding the location of the given feature in the first frame in the sequence of low resolution images; and

searching the second frame in the sequence of low resolution images for a corresponding image patch using a matching criterion.

17. The array camera of claim 16, wherein the matching criterion involves minimizing an error distance metric.

18. The array camera of claim 14, wherein the processor is further configured by software to perform feature detection within the high resolution image portions to identify high precision locations for said plurality of detected features by searching the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images using the high resolution image regions containing the features from the first frame in the low resolution sequence of images.

19. The array camera of claim 18, wherein the processor is further configured by software to search the high resolution image regions containing the features from the second frame in the sequence of low resolution images for features from the first frame in the sequence of low resolution images by comparing high resolution image regions containing features from the second frame in the sequence of low resolution images to the high resolution image portions containing the features from the first frame in the sequence of low resolution images using a matching criterion.

20. The array camera of claim 19, wherein the matching criterion involves minimizing an error distance metric.