SHIFT VECTOR RELIABILITY DETERMINING APPARATUS AND METHOD

• US 20130301928A1
• Filed: 05/03/2013
• Published: 11/14/2013
• Est. Priority Date: 05/11/2012
• Status: Abandoned Application

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Springer, Paul, Nishi, Toru, Bruggemann, Matthias
• Application Number:
  US13/887,021
• Publication Number:
  US 20130301928A1
• Time in Patent Office: Days
• US Class Current: 382/197
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

An apparatus for determining the reliability of shift vectors between two images comprises an image compensation unit for compensating local shifts between a first image and a second image and to obtain a compensated second image. A similarity estimation unit is provided for determining a similarity information by determining one or more similarity measures between said first image and said compensated second image. A vector consistency check device for comparing shift vectors describing the shift between said first image and said second image from different shift estimation directions to obtain a consistency weight information, and a combination unit for combining said similarity information and said consistency weight information to obtain a reliability information describing the reliability of said shift vectors are provided.

Claims

1. An apparatus for determining the reliability of shift vectors between two images comprising:

an image compensation unit configured to compensate local shifts between a first image and a second image and to obtain a compensated second image,

a similarity estimation device configured to determine a similarity information by determining one or more similarity measures between said first image and said compensated second image,

a vector consistency check device configured to compare shift vectors describing the shift between said first image and said second image from different shift estimation directions to obtain a consistency weight information, and

a combination unit configured to combine said similarity information (2) and said consistency weight information to obtain a reliability information describing the reliability of said shift vectors.

2. The apparatus as claimed in claim 1,

wherein said similarity estimation device comprises

an image analysis unit configured to analyse the first image and/or second image to obtain a feature information indicating one or more image features of the respective image and

an adaptive similarity estimation unit to adaptively determine one or more similarity measures depending on said feature information to obtain a first similarity information.

3. The apparatus as claimed in claim 2,

wherein said adaptive similarity estimation unit is configured to select the kind of similarity measure and/or to set parameters of an applied similarity measure depending on said feature information.

4. The apparatus as claimed in claim 1,

wherein said combination unit is a multiplication unit configured to pixel-wise or pixel-area-wise multiply said similarity information and said consistency weight information to obtain said reliability information.

5. The apparatus as claimed in claim 1,

wherein said vector consistency check device is configured to compare motion vectors describing the motion between said first image and said second image, a first motion vector describing the motion estimated from said first image to said second image and a second motion vector describing the motion estimated from said second image to said first image.

6. The apparatus as claimed claim 1,

wherein said vector consistency check device is configured to compare disparity vectors describing the disparity between said first image of a first view and said second image of a second view, a first disparity vector describing the disparity estimated from said first image to said second image and a second motion vector (V₂) describing the disparity estimated from said second image to said first image.

7. The apparatus as claimed in claim 1,

wherein said similarity estimation device comprises a non-adaptive similarity estimation unit (70)—

configured to non-adaptively determine one or more similarity measures independent from feature information to obtain a second similarity information.

8. The apparatus as claimed in claim 2,

wherein said image analysis unit comprises a contrast determination unit configured to determine a contrast information indicating the local contrast of an input image.

9. The apparatus as claimed in claim 2,

wherein said image analysis unit comprises a flat area detection device configured to determine a flatness information indicating flat and textured areas of an input image.

10. The apparatus as claimed in claim 9,

wherein said flat area detection device comprises

a gradient detection unit configured to determine the gradient in two directions, in particular two orthogonal directions, for an input image,

a gradient variance computing unit configured to compute the variance of the determined gradients, and

a flat area detection unit configured to determine said flatness information by use of a gradient variance threshold.

11. The apparatus as claimed in claim 9,

wherein said adaptive similarity estimation unit is configured to adaptively determine a normalized cross correlation weight factor from said first image and said compensated second image using said flatness information.

12. The apparatus as claimed in claim 8,

wherein said adaptive similarity estimation unit is configured to adaptively determine a summed absolute difference weight factor from said first image and said compensated second image using said flatness information and said contrast information (5a).

13. The apparatus as claimed in claim 8,

wherein said non-adaptive similarity estimation unit (70) is configured to determine a luminance difference weight factor (2b1) from said first image (X) and said compensated second image (Y) using said contrast information (5a).

14. The apparatus as claimed in claim 7,

wherein said non-adaptive similarity estimation unit is configured to determine a structural similarity weight factor from said first image and said compensated second image.

15. The apparatus as claimed in claim 1,

wherein said vector consistency check device is configured to compare the difference between said shift vectors to a shift threshold and to set said consistency weight information to a first or a second value depending on the result of said comparison.

16. The apparatus as claimed in claims 7,

wherein said combination unit is configured to combine said first similarity information, second similarity information and said consistency weight information to obtain said reliability information.

17. An apparatus for determining the reliability of shift vectors between two images comprising:

an image compensation means for compensating local shifts between a first image and a second image and to obtain a compensated second image,

a similarity estimation means for determining a similarity information by determining one or more similarity measures between said first image and said compensated second image,

a vector consistency check means for comparing shift vectors describing the shift between said first image and said second image from different shift estimation directions to obtain a consistency weight information, and

a combination means for combining said similarity information and said consistency weight information to obtain a reliability information describing the reliability of said shift vectors.

18. An image enhancement apparatus for enhancing an input image of a sequence of input images and obtaining an enhanced output image, said apparatus comprising

a shift apparatus for shifting one or more images by use of shift vectors between two images, and

an apparatus as claimed in claim 17 for determining the reliability of said shift vectors,

wherein said shift apparatus is configured to take said reliability into account when using said shift vector for shifting one or more images.

19. A method for determining the reliability of shift vectors between two images comprising:

compensating local shifts between a first image and a second image and to obtain a compensated second image, and

determining a similarity information by determining one or more similarity measures between said first image and said compensated second image,

comparing shift vectors describing the shift between said first image and said second image from different shift estimation directions to obtain a consistency weight information, and

combining said similarity information and said consistency weight information to obtain a reliability information describing the reliability of said shift vectors.

21. A non-transitory computer-readable recording medium that stores therein a computer program product, which, when executed by a processor, causes the method according to claim 19 to be performed.

20. (canceled)