Handling perspective magnification in optical flow processing

• US 10,621,446 B2
• Filed: 08/23/2017
• Issued: 04/14/2020
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: Texas Instruments, Inc.
• Original Assignee: Texas Instruments, Inc.
• Inventors: Garud, Hrushikesh Tukaram, Mathew, Manu, Nagori, Soyeb Noormohammed
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/684,321
• Publication Number:
  US 20180181816A1
• Time in Patent Office: 965 Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search

Abstract

A method of optical flow estimation is provided that includes identifying a candidate matching pixel in a reference image for a pixel in a query image, determining a scaled binary pixel descriptor for the pixel based on binary census transforms of neighborhood pixels corresponding to scaling ratios in a set of scaling ratios, determining a scaled binary pixel descriptor for the candidate matching pixel based on binary census transforms of neighborhood pixels corresponding to scaling ratios in the set of scaling ratios, and determining a matching cost of the candidate matching pixel based on the scaled binary pixel descriptors.

Claims

1. A system comprising:

a pixel match engine configured to;

identify a candidate matching pixel in a reference image for a pixel in a query image; and

determine a matching cost for the candidate matching pixel based on a first pixel descriptor for the pixel and a second pixel descriptor for the candidate matching pixel; and

a census transform engine configured to determine the first pixel descriptor and the second pixel descriptor, wherein to determine the first pixel descriptor, the census transform engine is configured to;

determine a first descriptor for the pixel, in which the determination of the first descriptor comprises a census transform of first neighborhood pixels in the query image;

determine a second descriptor for the pixel, in which the determination of the second descriptor comprises a census transform of second neighborhood pixels in the query image; and

determine the first pixel descriptor based in part on the first descriptor and the second descriptor; and

to determine the second pixel descriptor, the census transform engine is configured to;

determine a third descriptor for the candidate matching pixel, in which the determination of the third descriptor comprises a binary census transform of third neighborhood pixels in the reference image;

determine a fourth descriptor for the candidate matching pixel, in which the determination of the fourth descriptor comprises a binary census transform of fourth neighborhood pixels in the reference image; and

determine the second pixel descriptor based in part on the third descriptor and the fourth descriptor.

2. The system of claim 1, wherein the first neighborhood pixels and the second neighborhood pixels are comprised in a 5×

5 pixel neighborhood and the third neighborhood pixels and the fourth neighborhood pixels are comprised in a 7×

7 pixel neighborhood.

3. The system of claim 1, wherein the pixel match search engine is further configured to determine the matching cost as a Hamming distance of the first pixel descriptor and the second pixel descriptor.

4. The system of claim 1, wherein the pixel match search engine is further configured to determine the matching cost as an accumulated Hamming distance of a first accumulated binary pixel descriptor for the pixel comprising the first pixel descriptor and a second accumulated binary pixel descriptor for the candidate matching pixel comprising the second pixel descriptor, wherein the first accumulated binary pixel descriptor comprises pixel descriptors for pixels in a cost calculation window centered on the pixel and the second accumulated binary pixel descriptor comprises pixel descriptors for pixels in the cost calculation window centered on the candidate matching pixel.

5. The system of claim 1, wherein the pixel match engine is further configured to determine the matching cost as a scaled accumulated Hamming distance between pixel descriptors for pixels in a cost calculation window centered on the pixel and pixel descriptors for pixels in the cost calculation window centered on the candidate matching pixel, wherein the scaled accumulated Hamming distance is determined as per

6. The system of claim 1, wherein the system is comprised in a driver assistance system.

7. The system of claim 1, wherein the system is comprised in a hardware optical flow accelerator.

8. A method comprising:

identifying a candidate matching pixel in a reference image for a pixel in a query image;

determining a first descriptor for the pixel based in part on a binary census transform of first neighborhood pixels in the query image;

determining a second descriptor for the pixel based in prt on the binary census transform of second neighborhood pixels in the query image;

determining a first pixel descriptor based in part on the first descriptor and the second descriptor;

determining a third descriptor for the candidate matching pixel based in part on a binary census transform of third neighborhood pixels in the reference image;

determining a fourth descriptor for the candidate matching pixel based in part on a binary census transform of fourth neighborhood pixels in the reference image;

determining a second pixel descriptor based in part on the third descriptor and the fourth descriptor; and

determining a matching cost of the candidate matching pixel based on the first pixel descriptor and the second pixel descriptor.

9. The method of claim 8, wherein the first neighborhood pixels and the second neighborhood pixels are comprised in a 5×

5 pixel neighborhood and the third neighborhood pixels and the fourth neighborhood pixels are comprised in a 7×

7 pixel neighborhood.

10. The method of claim 8, wherein determining a matching cost further comprises determining the matching cost as a Hamming distance of the first pixel descriptor and the second pixel descriptor.

11. The method of claim 8, wherein determining the matching cost further comprises determining the matching cost as an accumulated Hamming distance of a first accumulated binary pixel descriptor for the pixel comprising the first pixel descriptor and a second accumulated binary pixel descriptor for the candidate matching pixel comprising the second pixel descriptor, wherein the first accumulated binary pixel descriptor comprises pixel descriptors for pixels in a cost calculation window centered on the pixel and the second accumulated binary pixel descriptor comprises pixel descriptors for pixels in the cost calculation window centered on the candidate matching pixel.

12. The method of claim 8, wherein determining the matching cost further comprises determining the matching cost as a scaled accumulated Hamming distance between pixel descriptors for pixels in a cost calculation window centered on the pixel and pixel descriptors for pixels in the cost calculation window centered on the candidate matching pixel, wherein the scaled accumulated Hamming distance is determined as per

13. A method of optical flow estimation, the method comprising:

identifying a candidate matching pixel in a reference image for a pixel in a query image; and

determining a matching cost of the candidate matching pixel based on a first neighborhood of pixels of the pixel and a second neighborhood of pixels of the candidate matching pixel and a set of scaling ratios, wherein relative positions of pixels in the first neighborhood of pixels and relative positions of pixels in the second neighborhood of pixels used in determining the matching cost are selected based on the set of scaling ratios.

14. The method of claim 13, wherein determining a matching cost further comprises:

determining the matching cost as a scaled accumulated Hamming distance between first binary pixel descriptors for the first neighborhood of pixels and second binary pixel descriptors for the second neighborhood of pixels, wherein the scaled accumulated Hamming distance is determined as per

15. The method of claim 14, wherein the first binary pixel descriptors and the second binary pixel descriptors are pixel descriptors.

16. The method of claim 13, wherein determining a matching cost further comprises:

determining the matching cost as per a scaled sum-of-absolute-distances (SAD), wherein the scaled SAD is determined as per

17. The method of claim 13, wherein determining a matching cost further comprises:

determining the matching cost as a Hamming distance of a first pixel descriptor and a second pixel descriptor, wherein the first pixel descriptor is determined based on the first neighborhood of pixels and the second pixel descriptor is determined based on the second neighborhood of pixels.

18. The method of claim 13, wherein determining a matching cost further comprises:

determining the matching cost as an accumulated Hamming distance of a first accumulated binary pixel descriptor for the pixel and a second accumulated binary pixel descriptor for the candidate matching pixel, wherein the first accumulated binary pixel descriptor comprises pixel descriptors for pixels in the first neighborhood of pixels and the second accumulated binary pixel descriptor comprises pixel descriptors for pixels in the second neighborhood of pixels.

19. The method of claim 13, wherein determining a matching cost further comprises:

determining the matching cost based on a first pixel descriptor and a second pixel descriptor, wherein the first pixel descriptor comprises binary pixel descriptors determined by a binary census transform in the first neighborhood of pixels and the second pixel comprises binary pixel descriptors determined by the binary census transform in the second neighborhood of pixels.